Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

1 vote302 views1 answer
Numpy complex number
Python
0 vote296 views1 answer
TypeError: can't pickle weakref objects site:stackoverflow.com
Python
0 vote330 views1 answer
QuizListView login required django
Python Django
0 vote280 views1 answer
How to use ttk themes
Python
1 vote274 views1 answer
Compile python to pyc
Python
1 vote320 views1 answer
Add background image in django uploaded file
Python Django
5 votes267 views3 answers
Django change password command line
Python
0 vote279 views2 answers
List to excel python
Python
0 vote274 views1 answer
Save request response json to file python
Python
0 vote291 views1 answer
How to apply logarithm in pandas dataframe
Python