Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

3 votes376 views2 answers
Python button click code
Python
2 votes314 views1 answer
@transactional annotation
Python
0 vote451 views1 answer
Drf not getting form
Python
1 vote345 views3 answers
Check if number is between two numbers python
Python
2 votes348 views1 answer
Comparing file content in python
Python
0 vote390 views1 answer
Pathlib path forward or back slahses
Python
1 vote360 views1 answer
Create django group
Python
0 vote346 views2 answers
Python decimal remove trailing zero
Python
0 vote317 views1 answer
Python rotate pdf pages
Python
1 vote339 views1 answer
Python twilio certificate error
Python