Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

0 vote368 views1 answer
Python code to remove file extension
Python
0 vote343 views1 answer
Adding new fields in ModelAdmin with fieldsets to edit user, and add_fieldsets whan creating a new user
Python
2 votes340 views2 answers
Csv manipulation python
Python
0 vote356 views1 answer
Django select_related multiple
Python Django
0 vote336 views1 answer
Pip install Parser
Python
2 votes389 views2 answers
Python defaultdict to dict
Python
30 votes355 views3 answers
Pandas add a total row to dataframe
Python
0 vote336 views1 answer
Selenium proxy python chrome
Python
0 vote319 views1 answer
IsADirectoryError: [Errno 21] Is a directory: '/'
Python
0 vote298 views1 answer
Unique id on duplicate ids values pandas
Python