Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

6 votes287 views1 answer
How to set the icon of the window in pygame
Python
0 vote282 views1 answer
Tkmessagebox not found
Python
3 votes260 views1 answer
Scan space seperated integers in python using map
Python
1 vote265 views1 answer
Split list into list of lists python on every n element
Python
6 votes254 views2 answers
Python reverse words and swap case
Python
1 vote249 views1 answer
How to print alternate numbers in python
Python
0 vote293 views1 answer
Python random select no replace
Python
234 votes466 views6 answers
Random in python
Python
0 vote250 views1 answer
Pandas dataframe froms string
Python
1 vote240 views1 answer
Python how to use a variable to trigger an event
Python