Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

3 votes217 views1 answer
Print items in object python
Python
1 vote249 views1 answer
Filter foreign fileds django_filters
Python Django
10 votes235 views1 answer
Pandas select columns where value is true
Python
1 vote306 views5 answers
How to find avrage python
Python
0 vote226 views1 answer
Example 1: Reset Index & Drop Old Index pandas
Python
33 votes236 views6 answers
Python create requirements.txt
Python
0 vote224 views1 answer
Python write error to file
Python
-1 vote240 views1 answer
Histogram image processing python
Python
0 vote230 views1 answer
How to print two values from a list in python
Python
7 votes252 views6 answers
How to create a function in python
Python