Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

2 votes215 views2 answers
Pdf to csv python
Python
1 vote199 views2 answers
Python character list to string
Python
13 votes208 views5 answers
Get index of item in list
Python
0 vote224 views1 answer
Adding extra fields to user model django
Python Django
0 vote199 views9 answers
Django recapcha
Python
0 vote193 views2 answers
How to add extra zeros after decimal in python
Python
0 vote210 views1 answer
Cross entropy
Python
0 vote212 views1 answer
Data parsing app python
Python
0 vote205 views1 answer
Pandas dataframe from multiple csv
Python
3 votes203 views2 answers
How to make python code faster
Python