Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

1 vote266 views1 answer
Write a python program to split a given dictionary of lists into list of dictionaries.
Python
0 vote309 views1 answer
Eia api python
Python
0 vote308 views1 answer
Tar: Exiting with failure status due to previous errors
Python
1 vote303 views1 answer
Print a to z in python
Python
6 votes310 views2 answers
How many nan in array python
Python
0 vote299 views1 answer
Customizable TKinter Buttons Python
Python
0 vote304 views1 answer
Write int in file python
Python
1 vote245 views1 answer
Polynomial in python
Python
0 vote333 views1 answer
How to convert a list to a string by newline python
Python
3 votes299 views3 answers
Convert pyqt5 ui file to python file
Python