Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

96 votes224 views6 answers
If substring not in string python
Python
1 vote218 views2 answers
Python save input to text file
Python
68 votes208 views6 answers
List length python
Python
26 votes206 views6 answers
If then else python
Python
2 votes269 views1 answer
Check if user log in flask
Python Flask
38 votes232 views6 answers
Datetime python
Python
6 votes263 views2 answers
Python pandas series to dataframe
Python
0 vote229 views1 answer
1d random walk in python stack exchange
Python
0 vote192 views1 answer
Zip used in python stack overflow
Python
1 vote231 views2 answers
Python how to print variable value
Python