Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

34 votes295 views5 answers
Pandas max columns
Python
6 votes312 views2 answers
Show all columns pandas jupyter notebook
Python
4 votes297 views5 answers
Epoch to datetime python
Python
0 vote281 views1 answer
Basic script
Python
0 vote281 views1 answer
Df add value at first index
Python
0 vote295 views1 answer
How to duplicate a row in python
Python
0 vote283 views1 answer
Pytorch reduce dataset dataloader
Python
0 vote286 views1 answer
Timeit command line
Python
0 vote310 views1 answer
Mechanize python #2
Python
0 vote284 views1 answer
Check if array is monotonic python
Python