Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

-1 vote233 views1 answer
Import csv from google drive python
Python
2 votes245 views2 answers
Remove all rows where one ccolumns egale to nan
Python
0 vote249 views2 answers
Ip regex python
Python
1 vote248 views1 answer
Python retry
Python
0 vote242 views1 answer
Returns the first row as a Row
Python
18 votes251 views6 answers
2d list in python
Python
1 vote263 views1 answer
Jinja2 template import html with as
Python
0 vote230 views1 answer
Rubik's cube solver python
Python
0 vote242 views2 answers
With open
Python
0 vote272 views1 answer
Numpy combinations of 5 bits
Python