Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

1 vote175 views1 answer
How does urllib.parse.urlsplit work in python
Python
38 votes175 views3 answers
Pil python install
Python
0 vote161 views1 answer
How to count number from 1 to 10 in python
Python
0 vote165 views6 answers
Python math
Python
0 vote172 views1 answer
Napalm cli
Python
2 votes186 views6 answers
Boolean in python
Python
0 vote193 views1 answer
Install mamba conda
Python
1 vote180 views1 answer
Round godot
Python
13 votes189 views2 answers
Delete database entry using name django
Python
2 votes176 views2 answers
Python greatest common divisor
Python