Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

11 votes346 views10 answers
Python print
Python
1 vote330 views1 answer
How to make table using python
Python
0 vote309 views1 answer
Pickle.load from gpu device to cpu
Python
3 votes347 views2 answers
ModuleNotFoundError: No module named 'decouple'
Python
0 vote409 views1 answer
Python os.getenv not working
Python
0 vote318 views1 answer
Python openstreetmap multiple latitude
Python
0 vote300 views1 answer
Python slice a list a specific amount of times
Python
0 vote329 views3 answers
Python matrix condensed to square
Python
30 votes330 views3 answers
Iterating index array python
Python
0 vote316 views7 answers
Convert code python to java
Python