Course Introduction

Course Resources

• Previous Versions
• Syllabus
• Textbook

Hello, and welcome to CIS 526 - Web Application Development, and also CC 515 - Full Stack Web Development. Even though these are two different courses in the catalog, they teach the same content and will use the same Canvas course. So, anywhere you see CIS 526 in this course, you can also mentally substitute CC 515 in its place.

My name is Nathan Bean, and I will be your instructor for this course. My contact information is shown here, and is also listed on the syllabus, and on the home page of the course on K-State Online. My email address is nhbean@ksu.edu, and it is the official method of communication for matters outside of this course, since it allows me to have a record of our conversations and respond when I’m available. I’ll also be available via the K-State Teams app and the K-State CS Discord server, so you can easily chat with me there.

For communication in this course, there are two basic methods that I recommend. For general questions about the course, content, getting help with labs, and any other items relevant to the course, I encourage you to use the Ed Discussion board accessible through Canvas. This allows all of us to communicate in a single space, and it also means that any questions I answer will immediately be available for the whole class. For personal issues, grading questions, or if you have something that is a “to-do” item for me, please email me directly.

Before we begin, I must give credit to Russell Feldhausen for expanding the content of this course and preparing many of the videos you’ll be watching - so you’ll be seeing his face as often (or perhaps more often) than mine.

For a brief overview of the course, there are a total of 8 modules of content, containing textbook pages, activities, tutorials, and more that you’ll complete. In addition, throughout the semester you’ll be working on a large-scale web application project which consists of 6 milestones. The modules are configured in K-State Canvas as gated modules, meaning that you must complete each item in the module in order before continuing. There will be one module due each week, and you may work ahead at your own pace. Finally, all work in this course must be completed and all labs graded by no later than May 3rd, 2024.

Looking ahead to the rest of this first module, you’ll see that there are a few more items to be completed before you can move on. In the next video, Russ will discuss a bit more information about navigating through this course on Canvas, using Codio, and using the videos posted on YouTube.

One thing I highly encourage each of you to do is read the syllabus for this course in its entirety, and let me know if you have any questions. My view is that the syllabus is a contract between me as your teacher and you as a student, defining how each of us should treat each other and what we should expect from each other. I have made a few changes to my standard syllabus template for this course, and those changes are clearly highlighted. Finally, the syllabus itself is subject to change as needed as we adapt to this new course layout and format, and all changes will be clearly communicated to everyone before they take effect.

The grading in this course is very simple. First, 15% of your grade consists of completing the short activities and quizzes scattered throughout the course. Another 35% of your grade consists of completing the interactive tutorials. Finally, 50% of your grade comes from completing the 6 project milestones throughout the semester. Also, notice that the final milestone is worth double the amount of points, so it is very important that you get to the end of the course and complete that milestone. There will be some extra credit points available, mainly through the Bug Bounty assignment, which you will review as part of this module. Lastly, the standard “90-80-70-60” grading scale will apply, though I reserve the right to curve grades up to a higher grade level at my discretion. Therefore, you will never be required to get higher than 90% for an A, but you may get an A if you score slightly below 90% if I choose to curve the grades.

Since this is a completely online course, you may be asking yourself what is different about this course. First off, you can work ahead at your own pace, and turn in work whenever you like before the due date. However, as discussed before, you must do all the readings and assignments in order before moving on, so you cannot skip ahead.

In addition, due to the flexible online format of this class, there won’t be any long lecture videos to watch. Instead, each module will consist of several short lessons and tutorials, each focused on a particular topic or task. Likewise, there won’t be any textbooks formally used, but you’ll be directed to a bevy of online resources for additional information.

What hasn’t changed, though, is the basic concept of a college course. You’ll still be expected to watch or read about 6 hours of content to complete each module. In addition to that, each lab assignment may require anywhere from 1 to 6 hours of work to complete. If you plan on doing a module every week, that roughly equates to 6 hours of content and 6 hours of homework each week, which is the expected workload from a 3 credit hour college course during the summer.

Also, while some of the activities will be graded automatically, much of the grading will still be done directly by me. This includes the project milestones. For each milestone, I’ll try to give you timely feedback so you can improve on your design before the next milestone is due.

For this class, each student is required to have access to a modern web browser and a high-speed internet connection. If you have any concerns about meeting these requirements, please contact me ASAP! We may have options available through some on-campus resources to help you out.

This summer, I’ll be working on a few updates to this course. These updates will mainly affect the second half of the course, and focus on updating a few of the items and introducing some newer technologies and libraries you may come across.

Finally, as you are aware, this course is always subject to change. While we have taught this class several times before, there may be a few hiccups as we get started due to new software and situations. The best advice I have is to look upon this graphic with the words “Don’t Panic” written in large, friendly letters, and remember that it’ll all work out in the end as long as you know where your towel is.

So, to complete this module, there are a few other things that you’ll need to do. The next step is to watch the video on navigating Canvas and using the YouTube videos, which will give you a good idea of how to most effectively work through the content in this course.

To get to that video, click the “Next” button at the bottom right of this page.

Navigating Canvas

YouTube Video

This course makes extensive use of several features of Canvas which you may or may not have worked with before. To give you the best experience in this course, this page will briefly describe those features and the best way to access them.

When you first access the course on Canvas, you will be shown this homepage, with my contact information and any important information about the course. This is a quick, easy reference for you if you ever need to get in touch with me.

Let’s walk through the options in the main menu to the left. First, any course announcements will be posted in the Announcements section, which is available here. Those announcements will also be configured to send emails to all students when they are posted, though in your personal Canvas settings you can disable email notifications if you so choose. Please make sure you check here often for any updates to course information.

The next section is Modules, which is where you’ll primarily interact with the course. You’ll notice that I’ve disabled several of the common menu items in this course, such as Files and Assignments. This is to simplify things for you as students, so you remember that all the course content is available in one place.

When you first arrive at the Modules section, you’ll see all of the content in the course laid out in order. If you like, you can minimize the modules you aren’t working on by clicking the arrow to the left of the module name.

As you look at each module, you’ll see that it gives quite a bit of information about the course. At the top of each module is an item telling you what parts of the module you must complete to continue. In this case, it says “Complete All Items.” Likewise, the following modules may list a prerequisite module, which you must complete before you can access it.

Within each module is a set of items, which must be completed in listed order. Under each item you’ll see information about what you must do in order to complete that item. For many of them, it will simply say “view,” which means you must view the item at least once to continue. Others may say “contribute,” “submit,” or give a minimum score required to continue. For assignments, it also helpfully gives the number of points available, and the due date.

Let’s click on the first item, Course Introduction, to get started. You’ve already been to this page by this point. Course pages will primarily consist of readings covering the content of the course. Some may also include an embedded video; in this case the video will be followed by slides and a downloadable version of the video, and a rough script for quick reference - as is the case for this page.

When you are ready to move to the next step in a module, click the “Next” button at the bottom of the page. Canvas will automatically add “Next” and “Previous” buttons to each piece of content which is accessed through the Modules section, which makes it very easy to work through the course content. I’ll click through a couple of items here.

At any point, you may click on the Modules link in the menu to the left to return to the Modules section of the site. You’ll notice that I’ve viewed the first few items in the first module, so I can access more items here. This is handy if you want to go back and review the content you’ve already seen, or if you leave and want to resume where you left off. Canvas will put green checkmarks to the right of items you’ve completed.

Finally, you’ll find the usual Canvas links to view your Grades in the course, as well as People listing instructors, TAs, and fellow students taking the course.

Where to Find Help

YouTube Video

Resources

• Slides
• K-State IT Help Desk - Email helpdesk@ksu.edu
• Syllabus
• K-State Online Canvas Help
• Instructure Canvas Guides
• Codio Documentation
• K-State Libraries
• K-State CS Support
• K-State CS Discord
• K-State CS Advising
• K-State Engineering Student Services
• K-State Office of Student Life
• K-State Report It

As you work on the materials in this course, you may run into questions or problems and need assistance. This video reviews the various types of help available to you in this course.

First and foremost, anytime you have a questions or need assistance in the course, please post in the course Discord room. It is the best place to go to get help with anything related to this course, from the tutorials and projects to issues with Codio and Canvas. Before you post on Discord, take a minute to look around and make sure the question has not already been posted before. It will save everyone quite a bit of time.

There are a few major reasons we’ve chosen to use Discord in this program. Our goal is to respond as quickly as possible, and having a single place for all questions allows the instructors and the TAs to work together to answer questions and solve problems quickly. As an added bonus, it reduces the amount of email generated by the class. Discord includes lots of features to make your messages easily readable using both markdown and code blocks. Finally, by participating in discussions on Discord and helping to answer questions from your fellow students, you can earn extra credit points!

Of course, as another step you can always exercise your information-gathering skills and use online search tools such as Google to answer your question. While you are not allowed to search online for direct solutions to assignments or projects, you are more than welcome to use Google to access programming resources such as the Mozilla Developer Network, Node language documentation, CSS-Tricks, StackOverflow, and other tutorials. I can definitely assure you that programmers working in industry are often using Google and other online resources to solve problems, so there is no reason why you shouldn’t start building that skill now.

If all else fails, please email me and let me know. Make sure you clearly explain your question and the steps you’ve taken to solve it thus far. If I feel it can be easily answered by one of the earlier steps, I may redirect you back to those before answering it directly. But, at times there are indeed questions that come up that don’t have an easy answer, and I’m more than happy to help answer them as needed.

Beyond Discord, there are a few resources you should be aware of. First, if you have any issues working with K-State Canvas, K-State IT resources, or any other technology related to the delivery of the course, your first source of help is the K-State IT Helpdesk. They can easily be reached via email at helpdesk@ksu.edu. Beyond them, there are many online resources for using Canvas, all of which are linked in the resources section below the video. As a last resort, you may also want to post in Discord, but in most cases we may simply redirect you to the K-State helpdesk for assistance.

Similarly, if you have any issues using the Codio platform, you are welcome to refer to their online documentation. Their support staff offers a quick and easy chat interface where you can ask questions and get feedback within a few minutes.

Next, we have grading and administrative issues. This could include problems or mistakes in the grade you received on a project, missing course resources, or any concerns you have regarding the course and the conduct of myself and your peers. Since this is an online course, you’ll be interacting with us on a variety of online platforms, and sometimes things happen that are inappropriate or offensive. There are lots of resources at K-State to help you with those situations. First and foremost, please DM me on Discord as soon as possible and let me know about your concern, if it is appropriate for me to be involved. If not, or if you’d rather talk with someone other than me about your issue, I encourage you to contact either your academic advisor, the CS department staff, College of Engineering Student Services, or the K-State Office of Student Life. Finally, if you have any concerns that you feel should be reported to K-State, you can do so at https://www.k-state.edu/report/. That site also has links to a large number of resources at K-State that you can use when you need help.

Finally, if you find any errors or omissions in the course content, or have suggestions for additional resources to include in the course, DM the instructors on Discord. There are some extra credit points available for helping to improve the course, so be on the lookout for anything that you feel could be changed or improved.

So, in summary, Discord should always be your first stop when you have a question or run into a problem. For issues with Canvas or Codio, you are also welcome to refer directly to the resources for those platforms. For questions specifically related to the projects, use Discord for sure. For grading questions and errors in the course content or any other issues, please PM the instructors on Discord for assistance.

Our goal in this program is to make sure that you have the resources available to you to be successful. Please don’t be afraid to take advantage of them and ask questions whenever you want.

What You'll Learn

The following is an outline of the topics we will be covering and when.

Week 1: Web Application Foundations

• The Document Object Model
  • Chapter 1
  • [Tutorial] Creating a Dialog
• HTTP
  • Chapter 2
  • [Activity] Making Manual HTTP Requests
• Responsive Design
  • Chapter B
  • [Tutorial] Responsive Card Layout
• Project Milestone 1
  • Card Layout
  • Accessing Data from a Web API

Week 2: Advanced JavaScript

• JSON and AJAX
  • [Activity] Working with JSON
  • [Activity] Making an AJAX Request
• Asynchronous JavaScript
  • Chapter 3
  • [Tutorial] Web Workers
• Introduction to Node
  • Chapter 4
  • [Tutorial] Your First Package
  • [Activity] Fun with Files
• Project Milestone 2
  • Request Data via AJAX
  • Dynamically Render Page

Week 3: Web Servers

• Basic Web Server
  • Chapter 5
  • [Tutorial] Hello Web
• File Server Basics
  • Chapter 5 (cont.)
  • [Tutorial] Node File Server
• Directory Listing
  • Chapter 5 (cont.)
  • [Tutorial] Directory Listing
• Partial Downloads
  • Chapter 5 (cont.)
  • Chapter C
  • [Activity] Regular Expressions
  • [Tutorial] Streaming Media
• Project Milestone 3
  • Refactor Project to Node

Week 4: Dynamic Web Servers

• Server Pages
  • Chapter 6
  • [Exercise] ECMAScript Server Pages
  • [Tutiroal] Image Gallery
• Uploading Data
  • Chapter 6 (cont.)
  • [Exercise] Uploading Form Data
  • [Exercise] Uploading Files
• Adding State
  • Chapter 6 (cont.)
  • [Activity] Fun with Cookies
  • [Tutorial] Gallery Favorites

Week 5: Full Stack Development

• Templates
  • Chapter 6 (cont.)
  • [Tutorial] Node Directory Listing in EJS
  • [Exercise] A Template By Another Name
• Full Stack Development
  • Chapter 6 (cont.)
• Persistent Storage
  • Chapter 7
  • [Tutorial] Blog Part 1
• Routing
  • Chapter 8
  • [Tutorial] Blog Part 2
  • [Tutorial] Blog Part 3
• Project Milestone 4
  • REfactor to Express
  • Add API Endpoints
  • Handle Requests for Items
  • Templates

Week 6: Web Frameworks

• Authentication
  • Chapter 9
  • [Tutorial] Blog Part 4
  • [Tutorial] Blog Part 5
• APIs
  • Chapter 8 (cont.)
  • [Tutorial] Blog Part 6
  • [Exercise] Web Hook Demo
• Web Frameworks
  • Chapter 10
  • [Tutorial] Single Page App Part 1
  • [Tutorial] Single Page App Part 2
• Project Milestone 5
  • Implement Authentication
  • Restrict Form Access to Authenticated Users

Week 7 & 8: Potpourri

• Single Sign On
  • Chapter 9 (cont.)
  • [Tutorial] CAS Authentication
• ASP.NET MVC
  • [Tutorial] ASP.NET MVC
• React & Websockets
  • [Tutorial] React Chat App
• Project Milestone 6
  • Mark Requests Complete
  • Administrator Role & Tasks

Course Textbooks

This course does not have a required print textbook. The resources presented in the modules are also organized into an online textbook that can be accessed here: https://textbooks.cs.ksu.edu/cis526. You may find this a useful reference if you prefer a traditional textbook layout. Additionally, since the textbook exists outside of Canvas’ access control, you can continue to utilize it after the course ends.

O’Reilly for Higher Education

If you are looking for additional resources to support your learning, a great resource that is available to Kansas State University students is the O’Reilley For Higher Education digital library offered through the Kansas State University Library. These include electronic editions of thousands of popular textbooks as well as videos and tutorials. As of this writing, a search for HTML returns 33,690 results, CSS 8,638 results, JavaScript 16,725 results, and Node.js 6,572 results.

There are likewise materials for other computer science topics you may have an interest in - it is a great resource for all your CS coursework. It costs you nothing (technically, your access was paid for by your tuition and fees), so you might as well make use of it!

Spring 2024 Syllabus

CIS 526 - Web Application Development

CC 515 - Full Stack Web Development

This syllabus covers both courses. They are taught using the same content.

Instructor Contact Information

• Instructor: Nathan Bean (nhbean AT ksu DOT edu)
  I use he/him pronouns. Feel free to share your own pronouns with me, and I’ll do my best to use them!
• Office: DUE 2216
• Phone: (785) 483-9264 (Call/Text)
• Website: https://nathanhbean.com
• In-Person Office Hours: Wednesday 1:20pm-3:30pm
• Virtual Office Hours: By appointment via Zoom.

Preferred Methods of Communication:

• Email: Email is the official method of communication for this course. Any emails sent to the instructor regarding this course should be answered within one class day.
• Ed Discussions: For short questions and discussions of course content and assignments, Ed Discussions is preferred since questions can be asked once and answered for all students. The Ed Discussion board can be accessed through the course navigation on Canvas. Students are encouraged to post questions there and use that space for discussion, and the instructor will strive to answer questions there as well.
• Phone/Text: Emergencies only! I will do my best to respond as quickly as I can.

GTA Contact Information

• GTA: Joshua Garcia (josh98 AT ksu DOT edu)
• Office: DUE 1118
• In-Person Office Hours: 9:30am-10:30am Tuesdays and Thursdays

Prerequisites

• CIS 526: CIS 501, CIS 560 (Prerequisite or Concurrent Enrollment), and either CC 120,CMST 135, or equivalent experience in HTML, CSS, and JavaScript with instructor permission.
• CC 515: CC 315, CC 410 (Prerequisite or Concurrent Enrollment), and either CC 120, CMST 135, or equivalent experience in HTML, CSS, and JavaScript with instructor permission.

Students may enroll in CIS or CC courses only if they have earned a grade of C or better for each prerequisite to those courses.

Course Overview

Fundamental principles and best practices of web development, user interface design, web API design, advanced web interfaces, web development frameworks, single-page web applications, web standards and accessibility issues.

Course Description

This course focuses on the creation of web applications - programs that use the core technologies of the world-wide-web to deliver interactive and dynamic user experiences. It builds upon a first course in authoring web pages using HTML/CSS/JavaScript, introduces the creation of web servers using the Node programming languages, and building sophisticated web clients using declarative component-based design frameworks like React.

Student Learning Outcomes

The following are the learning objectives of this course:

1. Students will develop a thorough understanding of the http client request - server response pattern, and be able to implement multiple kinds of requests and responses, including HTML tags, browser-based JavaScript, programmatically, and with tools.
2. Students will understand and be able to make use of asynchronous programming, including creating original asynchronous functions and utilizing promises and the async/await key words.
3. Students will be able to develop traditional full-stack web applications using Node, a SQL database, and a Linux OS.
4. Students will be able to develop client-side [progressive] web applications using transpilation and minimization.
5. Students will be able to develop secure web applications using password authentication, cookies, and json web tokens.

Major Course Topics

• The Document Object Model
• Responsive Web Design
• JavaScript Object Serialization Notation
• JavaScript Event Loop
• Asynchronous functions
• Promises
• async/await
• HTTP
• AJAX & Fetch
• Routing
• REST
• Form Serialization Formats
• Developing APIs
• Database Object Relational Mappers
• Template Rendering
• Single-page Applications
• Progressive Web Applications

Course Structure

This course is divided in modules, which typically consist of a series of lesson content (as video lectures or online textbook materials) followed by a hands-on tutorial. The tutorials show how to take the ideas just discussed in the lessons and apply them in creating web applications in a step-by-step manner. Following every third module is a larger project assignment, where you will utilize the skills you’ve been developing from the lessons and tutorials to iteratively create a web application.

For the eight-week summer course, each cluster of three modules plus project should be completed in one week. There is a lot to learn and much of the learning involved is hands-on as you work on the code for tutorials and your projects. It is recommended you set aside 10-20 hours per week to focus on this course.

The Work

There is no shortcut to becoming a great programmer or web developer. Only by doing the work will you develop the skills and knowledge to make you a successful web developer. This course is built around that principle, and gives you ample opportunity to do the work, with as much support as we can offer.

Lessons: Lessons are delivered in written or video (with written transcript) form. Sprinkled between lessons are activities and quizzes that check your understanding of the readings and lecture content.

Tutorials: Tutorials are delivered through Codio, and offer immediate, automatically generated feedback as you complete the assignments, letting you know if you’ve made a mistake or skipped a step. You can run these assessments as many times as needed until you have completed the project to your satisfaction.

Projects: The projects are more free-form - I want you to be able to flex your creative muscles and make a web app that both meets your customer’s needs and reflects your own style and approach. These will be graded by hand using a rubric that focuses on functionality, code quality, accessibility, and aesthetics.

Grading

In theory, each student begins the course with an A. As you submit work, you can either maintain your A (for good work) or chip away at it (for less adequate or incomplete work). In practice, each student starts with 0 points in the gradebook and works upward toward a final point total earned out of the possible number of points. IIn this course, each assignment constitutes a a portion of the final grade, as detailed below:

• 15% - Activities & Quizzes
• 35% - Tutorials
• 50% - Projects

Up to 5% of the total grade in the course is available as extra credit. See the Extra Credit - Bug Bounty and Extra Credit - Helping Hand assignments for details.

Letter grades will be assigned following the standard scale:

• 90% - 100% → A
• 80% - 89.99% → B
• 70% - 79.99% → C
• 60% - 59.99% → D*
• 50% - 0% → F

* Note that CS Majors must earn a C or better to use the CIS 526 course for their degree.

Submission, Regrading, and Early Grading Policy

As a rule, submissions in this course will not be graded until after they are due, even if submitted early. Students may resubmit assignments many times before the due date, and only the latest submission will be graded. For assignments submitted via GitHub release tag, only the tagged release that was submitted to Canvas will be graded, even if additional commits have been made. Students must create a new tagged release and resubmit that tag to have it graded for that assignment.

Once an assignment is graded, students are not allowed to resubmit the assignment for regrading or additional credit without special permission from the instructor to do so. In essence, students are expected to ensure their work is complete and meets the requirements before submission, not after feedback is given by the instructor during grading. However, students should use that feedback to improve future assignments and milestones.

For the programming project milestones, it is solely at the discretion of the instructor whether issues noted in the feedback for a milestone will result in grade deductions in a later milestones if they remain unresolved, though the instructor will strive to give students ample time to resolve issues before any additional grade deductions are made.

Likewise, students may ask questions of the instructor while working on the assignment and receive help, but the instructor will not perform a full code review nor give grading-level feedback until after the assignment is submitted and the due date has passed. Again, students are expected to be able to make their own judgments on the quality and completion of an assignment before submission.

That said, a student may email the instructor to request early grading on an assignment before the due date, in order to move ahead more quickly. The instructor’s receipt of that email will effectively mean that the assignment for that student is due immediately, and all limitations above will apply as if the assignment’s due date has now passed.

Collaboration Policy

In this course, all work submitted by a student should be created solely by the student without any outside assistance beyond the instructor and TA/GTAs. Students may seek outside help or tutoring regarding concepts presented in the course, but should not share or receive any answers, source code, program structure, or any other materials related to the course. Learning to debug problems is a vital skill, and students should strive to ask good questions and perform their own research instead of just sharing broken source code when asking for assistance.

That said, the field of web development requires the use of lots of online documentation and reference materials, and the point of the class is to learn how to effectively use those resources instead of “reinventing the wheel from scratch” in each assignment. Whenever content in an assignment is taken from an outside source, this should be noted somewhere in the assignment.

Late Work

In this class, there is a tremendous amount of new skills to develop in a short amount of time. Falling behind will jeopardize your chances of successfully completing the course. Trying to complete late assignments while also working on new material will also make it unlikely that you will retain what you are trying to learn. It is critical that you keep on-track.

Any work submitted and graded after the due date is subject to a deduction of 10% of the total points possible on the assignment for each day that the assignment is late. For example, if an assignment is due on a Friday and is submitted the following Tuesday, it will be subject to a reduction of 40% of the total points possible, or 10% for each class day it was late. These late penalties will be automatically entered by Canvas - contact the instructor if any grades appear to be incorrect.

These deductions will only be applied to grades above 50% of the total points on the assignment. So, if you scored higher than 50%, your grade will be reduced by the late penalty down to a minimum grade of 50%. If you scored lower than 50% on the assignment, no deductions will be applied.

However, even if a module is not submitted on time, it must still be completed before a student is allowed to begin the next module. So, students should take care not to get too far behind, as it may be very difficult to catch up.

All course work must be submitted, and all interactively graded materials must be graded with the instructor, on or before the last day of the semester in which the student is enrolled in the course in order for it to be graded on time. No late work will be accepted after that date.

If you have extenuating circumstances, please discuss them with the instructor as soon as they arise so other arrangements can be made. If you know you have upcoming events that will prevent you from completing work in this course, you should contact the instructor ASAP and plan on working ahead before your event instead of catching up afterwards. If you find that you are getting behind in the class, you are encouraged to speak to the instructor for options to catch up quickly.

Incomplete Policy

Students should strive to complete this course in its entirety before the end of the semester in which they are enrolled. However, since retaking the course would be costly and repetitive for students, we would like to give students a chance to succeed with a little help rather than immediately fail students who are struggling.

If you are unable to complete the course in a timely manner, please contact the instructor to discuss an incomplete grade. Incomplete grades are given solely at the instructor’s discretion. See the official K-State Grading Policy for more information. In general, poor time management alone is not a sufficient reason for an incomplete grade.

Unless otherwise noted in writing on a signed Incomplete Agreement Form, the following stipulations apply to any incomplete grades given in this course:

1. Students will be given 6 calendar weeks from the end of the enrolled semester’s finals week to complete the course
2. Students understand that access to instructor and GTA assistance may be limited after the end of an academic semester due to holidays and other obligations
3. If a student fails to resolve an incomplete grade after 6 weeks, they will be assigned an ‘F’ in the course. In addition, they will be dropped from any other courses which require the failed course as a prerequisite or corequisite.
4. For CC courses only:
   1. Students may receive at most two incompletes in Computational Core courses throughout their time in the program.
   2. Any modules in a future CC course which depend on incomplete work will not be accessible until the previous course is finished
      1. For example, if a student is given an incomplete in CC 210, then all modules in CC 310 will be inaccessible until CC 210 is complete

Recommended Texts & Supplies

To participate in this course, students must have access to a modern web browser and broadband internet connection. All course materials will be provided via Canvas and Codio. Modules may also contain links to external resources for additional information, such as programming language documentation.

In particular you are encouraged to use:

• Node Docs - THe documentation for the Nodejs platform and APIs.
• Mozilla Developer Network - A key reference explaining how browsers implement the web standards
• CSS-Tricks - A collection of guides and articles on using CSS to accomplish a variety of tasks
• w3c.org - The online home of the World-Wide-Web Consortium, the organization that sets web technology standards

This course offers an instructor-written textbook, which is broken up into a specific reading order and interleaved with activities and quizzes in the modules. It can also be directly accessed at https://textbooks.cs.ksu.edu/cis526.

Students who would like additional textbooks should refer to resources available on the O’Reilley For Higher Education digital library offered by the Kansas State University Library. These include electronic editions of popular textbooks as well as videos and tutorials.

Subject to Change

The details in this syllabus are not set in stone. Due to the flexible nature of this class, adjustments may need to be made as the semester progresses, though they will be kept to a minimum. If any changes occur, the changes will be posted on the Canvas page for this course and emailed to all students.

Academic Honesty

Kansas State University has an Honor and Integrity System based on personal integrity, which is presumed to be sufficient assurance that, in academic matters, one’s work is performed honestly and without unauthorized assistance. Undergraduate and graduate students, by registration, acknowledge the jurisdiction of the Honor and Integrity System. The policies and procedures of the Honor and Integrity System apply to all full and part-time students enrolled in undergraduate and graduate courses on-campus, off-campus, and via distance learning. A component vital to the Honor and Integrity System is the inclusion of the Honor Pledge which applies to all assignments, examinations, or other course work undertaken by students. The Honor Pledge is implied, whether or not it is stated: “On my honor, as a student, I have neither given nor received unauthorized aid on this academic work.” A grade of XF can result from a breach of academic honesty. The F indicates failure in the course; the X indicates the reason is an Honor Pledge violation.

For this course, a violation of the Honor Pledge will result in sanctions such as a 0 on the assignment or an XF in the course, depending on severity. Actively seeking unauthorized aid, such as posting lab assignments on sites such as Chegg or StackOverflow or asking another person to complete your work, even if unsuccessful, will result in an immediate XF in the course.

The Codio platform can perform automatic plagiarism detection by comparing submitted projects against other students’ submissions and known solutions. That information may be used to determine if plagiarism has taken place.

In this course, unauthorized aid broadly consists of giving or receiving code to complete assignments. This could be code you share with a classmate, code you have asked a third party to write for you, or code you have found online or elsewhere.

Authorized aid - which is not a violation of the honor policy - includes using the code snippets provided in the course materials, discussing strategies and techniques with classmates, instructors, TAs, and mentors. Additionally, you may use code snippets and algorithms found in textbooks and web sources if you clearly label them with comments indicating where the code came from and how it is being used in your project.

You should restrict your use of code libraries to those specified in the assignment description or approved by the instructor. You can ask for approval via Discord in the course channel, and if granted, this approval is valid for the entire class for the specified assignment.

Standard Syllabus Statements

Students with Disabilities

At K-State it is important that every student has access to course content and the means to demonstrate course mastery. Students with disabilities may benefit from services including accommodations provided by the Student Access Center. Disabilities can include physical, learning, executive functions, and mental health. You may register at the Student Access Center or to learn more contact:

• Manhattan/Olathe/Global Campus – Student Access Center
  • accesscenter@k-state.edu
  • 785-532-6441
• K-State Salina Campus – Julie Rowe; Student Success Coordinator
  • jarowe@k-state.edu
  • 785-820-7908

Students already registered with the Student Access Center please request your Letters of Accommodation early in the semester to provide adequate time to arrange your approved academic accommodations. Once SAC approves your Letter of Accommodation it will be e-mailed to you, and your instructor(s) for this course. Please follow up with your instructor to discuss how best to implement the approved accommodations.

Expectations for Conduct

All student activities in the University, including this course, are governed by the Student Judicial Conduct Code as outlined in the Student Governing Association By Laws, Article V, Section 3, number 2. Students who engage in behavior that disrupts the learning environment may be asked to leave the class.

Mutual Respect and Inclusion in K-State Teaching & Learning Spaces

At K-State, faculty and staff are committed to creating and maintaining an inclusive and supportive learning environment for students from diverse backgrounds and perspectives. K-State courses, labs, and other virtual and physical learning spaces promote equitable opportunity to learn, participate, contribute, and succeed, regardless of age, race, color, ethnicity, nationality, genetic information, ancestry, disability, socioeconomic status, military or veteran status, immigration status, Indigenous identity, gender identity, gender expression, sexuality, religion, culture, as well as other social identities.

Faculty and staff are committed to promoting equity and believe the success of an inclusive learning environment relies on the participation, support, and understanding of all students. Students are encouraged to share their views and lived experiences as they relate to the course or their course experience, while recognizing they are doing so in a learning environment in which all are expected to engage with respect to honor the rights, safety, and dignity of others in keeping with the K-State Principles of Community.

If you feel uncomfortable because of comments or behavior encountered in this class, you may bring it to the attention of your instructor, advisors, and/or mentors. If you have questions about how to proceed with a confidential process to resolve concerns, please contact the Student Ombudsperson Office. Violations of the student code of conduct can be reported using the Code of Conduct Reporting Form. You can also report discrimination, harassment or sexual harassment, if needed.

Netiquette

Online communication is inherently different than in-person communication. When speaking in person, many times we can take advantage of the context and body language of the person speaking to better understand what the speaker means, not just what is said. This information is not present when communicating online, so we must be much more careful about what we say and how we say it in order to get our meaning across.

Here are a few general rules to help us all communicate online in this course, especially while using tools such as Canvas or Discord:

• Use a clear and meaningful subject line to announce your topic. Subject lines such as “Question” or “Problem” are not helpful. Subjects such as “Logic Question in Project 5, Part 1 in Java” or “Unexpected Exception when Opening Text File in Python” give plenty of information about your topic.
• Use only one topic per message. If you have multiple topics, post multiple messages so each one can be discussed independently.
• Be thorough, concise, and to the point. Ideally, each message should be a page or less.
• Include exact error messages, code snippets, or screenshots, as well as any previous steps taken to fix the problem. It is much easier to solve a problem when the exact error message or screenshot is provided. If we know what you’ve tried so far, we can get to the root cause of the issue more quickly.
• Consider carefully what you write before you post it. Once a message is posted, it becomes part of the permanent record of the course and can easily be found by others.
• If you are lost, don’t know an answer, or don’t understand something, speak up! Email and Canvas both allow you to send a message privately to the instructors, so other students won’t see that you asked a question. Don’t be afraid to ask questions anytime, as you can choose to do so without any fear of being identified by your fellow students.
• Class discussions are confidential. Do not share information from the course with anyone outside of the course without explicit permission.
• Do not quote entire message chains; only include the relevant parts. When replying to a previous message, only quote the relevant lines in your response.
• Do not use all caps. It makes it look like you are shouting. Use appropriate text markup (bold, italics, etc.) to highlight a point if needed.
• No feigning surprise. If someone asks a question, saying things like “I can’t believe you don’t know that!” are not helpful, and only serve to make that person feel bad.
• No “well-actually’s.” If someone makes a statement that is not entirely correct, resist the urge to offer a “well, actually…” correction, especially if it is not relevant to the discussion. If you can help solve their problem, feel free to provide correct information, but don’t post a correction just for the sake of being correct.
• Do not correct someone’s grammar or spelling. Again, it is not helpful, and only serves to make that person feel bad. If there is a genuine mistake that may affect the meaning of the post, please contact the person privately or let the instructors know privately so it can be resolved.
• Avoid subtle -isms and microaggressions. Avoid comments that could make others feel uncomfortable based on their personal identity. See the syllabus section on Diversity and Inclusion above for more information on this topic. If a comment makes you uncomfortable, please contact the instructor.
• Avoid sarcasm, flaming, advertisements, lingo, trolling, doxxing, and other bad online habits. They have no place in an academic environment. Tasteful humor is fine, but sarcasm can be misunderstood.

As a participant in course discussions, you should also strive to honor the diversity of your classmates by adhering to the K-State Principles of Community.

Discrimination, Harassment, and Sexual Harassment

Kansas State University is committed to maintaining academic, housing, and work environments that are free of discrimination, harassment, and sexual harassment. Instructors support the University’s commitment by creating a safe learning environment during this course, free of conduct that would interfere with your academic opportunities. Instructors also have a duty to report any behavior they become aware of that potentially violates the University’s policy prohibiting discrimination, harassment, and sexual harassment, as outlined by PPM 3010.

If a student is subjected to discrimination, harassment, or sexual harassment, they are encouraged to make a non-confidential report to the University’s Office for Institutional Equity (OIE) using the online reporting form. Incident disclosure is not required to receive resources at K-State. Reports that include domestic and dating violence, sexual assault, or stalking, should be considered for reporting by the complainant to the Kansas State University Police Department or the Riley County Police Department. Reports made to law enforcement are separate from reports made to OIE. A complainant can choose to report to one or both entities. Confidential support and advocacy can be found with the K-State Center for Advocacy, Response, and Education (CARE). Confidential mental health services can be found with Lafene Counseling and Psychological Services (CAPS). Academic support can be found with the Office of Student Life (OSL). OSL is a non-confidential resource. OIE also provides a comprehensive list of resources on their website. If you have questions about non-confidential and confidential resources, please contact OIE at equity@ksu.edu or (785) 532–6220.

Academic Freedom Statement

Kansas State University is a community of students, faculty, and staff who work together to discover new knowledge, create new ideas, and share the results of their scholarly inquiry with the wider public. Although new ideas or research results may be controversial or challenge established views, the health and growth of any society requires frank intellectual exchange. Academic freedom protects this type of free exchange and is thus essential to any university’s mission.

Moreover, academic freedom supports collaborative work in the pursuit of truth and the dissemination of knowledge in an environment of inquiry, respectful debate, and professionalism. Academic freedom is not limited to the classroom or to scientific and scholarly research, but extends to the life of the university as well as to larger social and political questions. It is the right and responsibility of the university community to engage with such issues.

Campus Safety

Kansas State University is committed to providing a safe teaching and learning environment for student and faculty members. In order to enhance your safety in the unlikely case of a campus emergency make sure that you know where and how to quickly exit your classroom and how to follow any emergency directives. Current Campus Emergency Information is available at the University’s Advisory webpage.

Student Resources

K-State has many resources to help contribute to student success. These resources include accommodations for academics, paying for college, student life, health and safety, and others. Check out the Student Guide to Help and Resources: One Stop Shop for more information.

Student Academic Creations

Student academic creations are subject to Kansas State University and Kansas Board of Regents Intellectual Property Policies. For courses in which students will be creating intellectual property, the K-State policy can be found at University Handbook, Appendix R: Intellectual Property Policy and Institutional Procedures (part I.E.). These policies address ownership and use of student academic creations.

Mental Health

Your mental health and good relationships are vital to your overall well-being. Symptoms of mental health issues may include excessive sadness or worry, thoughts of death or self-harm, inability to concentrate, lack of motivation, or substance abuse. Although problems can occur anytime for anyone, you should pay extra attention to your mental health if you are feeling academic or financial stress, discrimination, or have experienced a traumatic event, such as loss of a friend or family member, sexual assault or other physical or emotional abuse.

If you are struggling with these issues, do not wait to seek assistance.

• Kansas State University Counseling and Psychological Services offers free and confidential services to assist you to meet these challenges.
• Lafene Health Center has specialized nurse practitioners to assist with mental health.
• The Office of Student Life can direct you to additional resources.
• K-State Family Center offers individual, couple, and family counseling services on a sliding fee scale.
• Center for Advocacy, Response, and Education (CARE) provides free and confidential assistance for those in our K-State community who have been victimized by violence.

For Kansas State Salina Campus:

• Kansas State Salina Counseling Services offers free and confidential services to assist you to meet these challenges.
• The Kansas State Salina Office of Student Life can direct you to additional resources.
• The Kansas State Salina Campus offers several services for students, including health services, counseling, and academic assistance.

For Global Campus/K-State Online:

• K-State Online students have free access to mental health counseling with My SSP - 24/7 support via chat and phone.
• The Office of Student Life can direct you to additional resources.

University Excused Absences

K-State has a University Excused Absence policy (Section F62). Class absence(s) will be handled between the instructor and the student unless there are other university offices involved. For university excused absences, instructors shall provide the student the opportunity to make up missed assignments, activities, and/or attendance specific points that contribute to the course grade, unless they decide to excuse those missed assignments from the student’s course grade. Please see the policy for a complete list of university excused absences and how to obtain one. Students are encouraged to contact their instructor regarding their absences.

Copyright Notice

© The materials in this online course fall under the protection of all intellectual property, copyright and trademark laws of the U.S. The digital materials included here come with the legal permissions and releases of the copyright holders. These course materials should be used for educational purposes only; the contents should not be distributed electronically or otherwise beyond the confines of this online course. The URLs listed here do not suggest endorsement of either the site owners or the contents found at the sites. Likewise, mentioned brands (products and services) do not suggest endorsement. Students own copyright to what they create.

Original content in the course textbook at https://textbooks.cs.ksu.edu/cis526 is licensed under a Creative Commons BY-SA license by Nathan Bean unless otherwise stated.

Introduction

At this point, you should be familiar with the big three technologies of the world-wide-web HTML, CSS, and JavaScript (Feel free to visit the appendices for a quick review). These three technologies work together to create the web pages you interact with every day. Each has a role to play in defining the final appearance of a web page:

• Hyper-Text Markup Language (HTML) provides the structure and content
• Cascading Style Sheets (CSS) determine how that content should appear visually
• JavaScript provides interactivity, allowing both the structure and appearance of the page to change dynamically

We often refer to this division of responsibility as the separation of concerns. By placing all responsibility for appearance on a CSS file, we can refresh the look of a web application simply by replacing the old CSS file with a new one. Similarly, we can create a new page in our site that looks and feels like the rest of the site by creating a new HTML page that links to the site’s existing CSS files.

While you have written HTML, CSS, and JavaScript files in your prior learning experiences, you might not have thought about just how these files are processed, and those styling rules applied. In this chapter we will explore this topic in detail, while introducing some more advanced uses for each.

Document Object Model

The Document Object Model (or DOM) is a data structure representing the content of a web page, created by the browser as it parses the website. The browser then makes this data structure accessible to scripts running on the page. The DOM is essentially a tree composed of objects representing the HTML elements and text on the page.

Consider this HTML:

<!DOCTYPE html>
<html>
    <head>
        <title>Hello DOM!</title>
        <link href="site.css"/>
    </head>
    <body>
        <div class="banner">
            <h1>Hello DOM!</h1>
            <p>
                The Document Object Model (DOM) is a programming API for HTML and XML documents. It defines the logical structure of documents and the way a document is accessed and manipulated. In the DOM specification, the term "document" is used in the broad sense - increasingly, XML is being used as a way of representing many different kinds of information that may be stored in diverse systems, and much of this would traditionally be seen as data rather than as documents. Nevertheless, XML presents this data as documents, and the DOM may be used to manage this data.
            </p>
            <a href="https://www.w3.org/TR/WD-DOM/introduction.html">From w3.org's What is the Document Object Model?</a>
        </div>
    </body>
<html>

When it is parsed by the browser, it is transformed into this tree:

The DOM Tree and Developer Tools

Most browsers also expose the DOM tree through their developer tools. Try opening the example page in Chrome or your favorite browser using this link.

Now open the developer tools for your browser:

• Chrome CTRL + SHIFT + i or right-click and select ‘Inspect’ from the context menu.
• Edge CTRL + SHIFT + i or right-click and select ‘Inspect element’ from the context menu.
• Firefox CTRL + SHIFT + i or right-click and select ‘Inspect Element’ from the context menu.
• Safari Developer tools must be enabled. See the Safari User Guide

You should see a new panel open in your browser, and under its ’elements’ tab the DOM tree is displayed:

Collapsed nodes can be expanded by clicking on the arrow next to them. Try moving your mouse around the nodes in the DOM tree, and you’ll see the corresponding element highlighted in the page. You can also dynamically edit the DOM tree from the elements tab by right-clicking on a node.

Try right-clicking on the <h1> node and selecting ’edit text’. Change the text to “Hello Browser DOM”. See how it changes the page?

The page is rendered from the DOM, so editing the DOM changes how the page appears. However, the initial structure of the DOM is derived from the loaded HTML. This means if we refresh the page, any changes we made to the DOM using the developer tools will be lost, and the page will return to its original state. Give it a try - hit the refresh button.

You’ve now seen how the browser creates the DOM tree by parsing the HTML document and that DOM tree is used to render the page. Next, we’ll look at how styles interact with the DOM to modify how it is displayed.

CSS and the DOM

Cascading style sheets (CSS) also interact with the DOM. Consider this CSS code:

.banner {
    border: 4px solid gold;
    border-radius: 5rem;
    background-color: goldenrod;
    padding: 5rem;
    color: brown;
}

.banner > h1 {
    font-style: italic;
}

.banner p {
    text-decoration: line-through;
    font-size: 1.2rem;
}

When it is placed in the site.css file referenced by the HTML we discussed in the last section, the rules it defines are evaluated in terms of the DOM tree. The result is the page now looks like this:

Now let’s talk about how this CSS code and the DOM interact.

Consider the selector .banner. It looks for any element whose class attribute includes the string "banner". Hence, it matches the <div> element, adding a color, background color, border, and padding. This visualization of the DOM tree indicates the selected node in yellow:

Notice how the text in both the <h1> element and the <p> element are a reddish color? That color is the one defined in the .banner rule: color: #754907. The rule applies not only to the selected node, but to all its descendants in the DOM tree. This is the ‘cascading’ part of cascading style sheets - the rules flow down the DOM tree until they are overridden by more specific css rules in descendant nodes.

The second way CSS interacts with the DOM tree is through the CSS selectors themselves.

For example, the selector .banner > h1 uses the child combinator - the > symbol between .banner and h1. This indicates that the rule will be applied to any <h1> nodes that are direct children of the node with the class of "banner". As we can see from the DOM tree, the sole <h1> tag is a child of the <div.banner> node, so this rule is applied to it:

Similarly, the .banner p tag uses the descendant combinator - the space between the .banner and p. This indicates that the rule will be applied to any <p> nodes that are descended from the node with the class of "banner". This will apply no matter how far down the tree those nodes appear. Thus, if we added more <p> elements inside of a <div> that was a child of the <div.banner> node, it would apply to them as well.

You can see the example with the styling applied by following this link. Try adding the div and two paragraphs with the developer tools. See how the styles are automatically applied to the new nodes in the DOM tree?

CSS and Developer Tools

Speaking of developer tools, there is another handy tab that deals with CSS, the ‘Styles’ tab. It displays all the css rules applied to a specific node. Select one of your new <p> elements. Notice how the styles tab shows the css rule .banner p that we’ve been discussing? Moreover, it tells you which CSS file and which line in that file the rule is found on.

If you scroll down, it also shows you the rules inherited (cascaded) from the .banner rule:

If you scroll clear to the bottom, you will also see a visualization of the box model as it is applied to this element.

This can be very handy for debugging margin/padding/border issues.

Now scroll back up to the top of the styles tab. Notice the element.style {} rule? This displays inline CSS on the element, and we can also add our own inline CSS directly from the developer tools. Add the property key/value pair text-decoration: none. Notice what happens to the paragraph’s text? Also, notice how the now overridden rule has been struck through in the rule below.

This can be very handy for quickly trying out different style combinations without needing to add them to the CSS file. Much like changes to the DOM, these are not saved - refresh the page and they are gone.

Finally, notice that when the mouse hovers over a CSS property key/value pair, a checkbox appears next to it? If you uncheck the box, the property ceases to be applied. This can also be very handy for debugging CSS problems.

Now that we’ve seen how CSS interacts with the DOM tree, it’s time to turn our attention to the third web technology - JavaScript.

JavaScript and the DOM

The DOM tree is also accessible from JavaScript running in the page. It is accessed through the global window object, i.e. window.document or document.

Let’s use the ‘Console’ tab of the developer tools to access this object. Open the previous example page again from this link. Click the console tab to open the expanded console, or use the console area in the bottom panel of the elements tab:

With the console open, type:

> document

Once you hit the enter key, the console will report the value of the expression document, which exposes the document object. You can click the arrow next to the #document to expose its properties:

The document is an instance of the Document class. It is the entry point (and the root) of the DOM tree data structure. It also implements the Node and EventTarget interfaces, which we’ll discuss next.

The Node Interface

All nodes in the DOM tree implement the Node interface. This interface provides methods for traversing and manipulating the DOM tree. For example, each node has a property parentElement that references is parent in the DOM tree, a property childNodes that returns a NodeList of all the Node’s children, as well as properties referencing the firstChild, lastChild, previousSibling, and nextSibling.

Let’s try walking the tree manually. In the console, type:

> document.body.firstElementChild.firstElementChild

The body property of the document directly references the <body> element of the webpage, which also implements the Node interface. The firstElementChild references the first HTML element that is a child of the node, so in using that twice, we are drilling down to the <h1> element.

The EventTarget Interface

Each node in the DOM tree also implements the EventTarget interface. This allows arbitrary events to be attached to any element on the page. For example, we can add a click event to the <h1> element. In the console, type:

> document.body.firstElementChild.firstElementChild.addEventListener('click', function(e){
    console.log(e.target + ' clicked!');
});

The first argument to EventTarget.addEventListener is the event to listen for, and the second is a function to execute when the event happens. Here we’ll just log the event to the console.

Now try clicking on the Hello DOM! <h1> element. You should see the event being logged:

We can also remove event listeners with EventTarget.removeEventListener and trigger them programmatically with EventTarget.dispatchEvent.

Querying the DOM

While we can use the properties of a node to walk the DOM tree manually, this can result in some very ugly code. Instead, the Document object provides a number of methods that allow you to search the DOM tree for a specific value. For example:

• Document.getElementsByTagName returns a list of elements with a specific tag name, i.e. document.getElementsByTagName('p') will return a list of all <p> elements in the DOM.
• Document.getElementsByClassName returns a list of elements with a specific name listed in its class attribute, i.e. document.getElementsByClassName('banner') will return a list containing the <div.banner> element.
• Document.getElementById returns a single element with the specified id attribute.

In addition to those methods, the Document object also supplies two methods that take a CSS selector. These are:

• Document.querySelector which returns the first matching element in the DOM.
• Document.querySelectorAll which returns a list of all matching elements.

Let’s try selecting the <h1> tag using the querySelector method:

> var header = document.querySelector('h1');

Much easier than document.body.firstElementChild.firstElementChild isn’t it?

Manipulating DOM Elements

All HTML elements in the DOM also implement the HTMLElement interface, which also provides access to the element’s attributes and styling. So when we retrieve an element from the DOM tree, we can modify these.

Let’s tweak the color of the <h1> element we saved a reference to in the header variable:

> header.style.color = 'blue';

This will turn the header blue:

All of the CSS properties can be manipulated through the style property.

In addition, we can access the element’s classList property, which provides an add() and remove() methods that add/remove class names from the element. This way we can define a set of CSS properties for a specific class, and turn that class on and off for an element in the DOM tree, effectively applying all those properties at once.

Modifying the DOM

We can create new elements with the Document.createElement method. It takes the name of the new tag to create as a string, and an optional options map (a JavaScript object). Let’s create a new <p> tag. In the console:

> var p = document.createElement('p');

Now let’s give it some text:

> p.textContent = "Tra-la-la";

Up to this point, our new <p> tag isn’t rendered, because it isn’t part of the DOM tree. But we can use the Node.appendChild method to add it to an existing node in the tree. Let’s add it to the <div.banner> element. Type this command into the console:

document.querySelector('div.banner').appendChild(p);

As soon as it is appended, it appears on the page:

Note too that the CSS rules already in place are automatically applied!

$('p').text('Tra-la-la').appendTo('div.banner');

JavaScript Events

It should be no surprise that JavaScript features events - after all, we’ve already seen how the EventTarget interface allows us to attach event listeners to elements in the DOM tree. What might not be clear yet is how events are handled by JavaScript. JavaScript uses an event loop to process events. This is similar to Windows and other operating systems also handle events.

An event loop expressed in code looks something like:

function main
    initialize()
    while message != quit
        message := get_next_message()
        process_message(message)
    end while
end function

It’s basically an infinite loop that responds to messages, one message at a time. It might be more useful to see a visual representation:

Here we see not just the event loop, but also the event queue. This is a queue that holds events until the event loop is ready to process them. It works like the first-in-first-out queues you built in your data structures course (although it may also consider priorities of events).

On the far right are some common sources for JavaScript events - user input, the network, and timers. These are often managed by the operating system, and with modern multiple-processor computers can happen concurrently, i.e. at the same time. This is one reason the queue is so important - it allows JavaScript to process the events one at a time.

When the JavaScript VM has finished executing its current work, it pulls the next event from the event queue. This event is processed by the corresponding event listener function that either 1) you wrote, or 2) is the default action. If neither exists, the event is discarded.

An Example

Consider when the user clicks on a link on your page, let’s say <a id="demo" href="https://google.com">Google it!</a>. This creates a ‘click’ event for the <a> tag clicked on. Now let’s assume you’ve written an event handler and attached it to that anchor tag:

document.getElementById('demo').addEventListener('click', function(e) {
    e.preventDefault();
    alert('You clicked the "Google it!" link.');
});

The anonymous function function(e) {...} attached to the <a>’s ‘click’ event is invoked, with the event details being passed as the parameter e. Anchor tags have a default behavior - they open the linked page. So the line e.preventDefault(); informs JavaScript not to use this default behavior. Then we trigger an alert with the string 'You clicked the "Google it!" link.'.

If we hadn’t attached the event listener, then the page would have used the default response - loading a new page into the browser in the place of our current page.

If we clicked on an element that didn’t have a default action (like a <p> element) and you haven’t attached a listener the event is discarded and does nothing.

Concurrency in JavaScript

An important takeaway from the discussion of the event loop is that the actual processing of JavaScript code is always single-threaded. This avoids many of the common challenges of multi-threaded code. You don’t need to create semaphores, locks, and other multi-threading synchronization tools as your code will always be executing in a single thread.

However, JavaScript does retain many of the benefits of concurrency within its model. For example, when the DOM is loading and encounters an element referencing an external resource (i.e. a video, img, link, or script element), it triggers a request to retrieve that resource through the browser. The browser does so while the JavaScript code continues executing. When the resource is fully downloaded by the browser, it creates a 'load' event with the details, and adds it to the JavaScript event queue. Multiple files are therefore downloaded concurrently, but our JavaScript handles the results one-by-one in a single-threaded manner.

Think of the JavaScript event loop as a busy manager that only works on one thing at a time. The manager might send several workers out to get information. When they return, they form a line in front of the manager’s desk and wait patiently. Once the manager is finished with the task they have been working on, they take the report from the first worker in line, and starts doing what is needed to be done with the returned information. Once the manager finishes that, the next employee will report, and so on.

Common Events

There are many kinds of events in JavaScript; you can find a complete list in the MDN docs. However some of the ones you will likely find yourself using are:

• load - Triggered when a resource (i.e. an image, video, stylesheet, script) has finished loading. You can also listen for the load event on the document itself; here it will be triggered after all the resources on the page are loaded.

• change The value of an <input>, <textarea>, or <select> has changed

• focus triggered when an input gains focus (is the currently selected input)

• blur triggered when an input loses focus

• click The primary mouse button was clicked. On old browsers this might trigger for any button

• contextmenu The right mouse button was clicked

• mousedown A mouse button was pressed

• mouseup A mouse button was released

Timers

Timers play a special role in JavaScript’s concurrency model, and in many ways behave like events. For example, to cause the phrase “Hello time!” to be logged to the console in three seconds, you would write the code:

setTimeout(function() { console.log("Hello time!")}, 3000);

You will notice that the setTimeout() method takes a function to execute at that future point in time, much like attaching an event handler. The second argument is the number of milliseconds in the future for this event to occur. The timer works like an event, when the time expires, a corresponding event is added to the event queue, to trigger the delayed function.

An important side-effect of this approach is that you only know the timer’s result won’t happen before the delay you specify, but if the JavaScript virtual machine is engaged in a long-running process, it may be longer before your timer event is triggered.

For events you need to do on a regular interval, use setInterval() instead. This will invoke the supplied function at each elapsing of the supplied interval. It also returns a unique id that can be supplied to clearInterval() to stop the timed event.

setTimeout(doSomething, 0);

doSomething();

The DOM and External Resource Loading

One of the important aspects of working with HTML is understanding that an HTML page is more than just the HTML. It also involves a collection of resources that are external to the HTML document, but displayed or utilized by the document. These include elements like <link>, <script>, <video>, <img>, and <source> with src or href attributes set.

As the DOM tree is parsed and loaded and these external resources are encountered, the browser requests those resources as well. Modern browsers typically make these requests in parallel for faster loading times.

Once the HTML document has been completely parsed, the window triggers the DOMContentLoaded event. This means the HTML document has been completely parsed and added to the DOM tree. However, the external resources may not have all been loaded at this point.

Once those resources are loaded, a separate Load event is triggered, also on the window.

Thus, if you have JavaScript that should only be invoked after the page has fully loaded, you can place it inside an event listener tied to the load event, i.e.:

window.addEventListener('load', function(event) {
    // TODO: Add your code here ...
});

Or, if you want it invoked after the DOM is fully parsed, but external resources may still be loading:

window.addEventListener('DOMContentLoaded', function(event) {
    // TODO: Add your code here ...
});

The former - waiting for all resources to load - tends to be the most common. The reason is simple, if you are loading multiple JavaScript files, i.e. a couple of libraries and your own custom code, using the 'load' event ensures they have all loaded before you start executing your logic.

Consider the popular JQuery library, which provides shorthand methods for querying and modifying the DOM. It provides a JQuery() function that is also aliased to the $. The JQuery code to show a popup element might be:

$('#popup').show();

But if the JQuery library isn’t loaded yet, the $ is not defined, and this logic will crash the JavaScript interpreter. Any remaining JavaScript will be ignored, and your page won’t work as expected. But re-writing that to trigger after all resources have loaded, i.e.:

window.addEventListener('load', function(event) {
    // This code only is executed once all resources have been loaded
    $('#popup').show();
});

Ensures the JQuery library is available before your code is run.

// Using the onload property
window.onload = function(event) {...}

// Using onload property and lambda syntax 
window.onload = (event) => {...}

// Using the addEventListener and lambda syntax
window.addEventListener('load', (event) => {
    ...
});

// Using the JQuery library 
JQuery(function(){...});

// Using the JQuery library with lambda syntax 
JQuery(() => {...});

// Using the JQuery library with $ alias
$(function(){...});

Loading Resources after the DOM is loaded

There are really two ways to load resources into an HTML page from your JavaScript code. One is indirect, and the other direct. The indirect method simply involves creating DOM elements linked to an outside resource, i.e.:

var image = document.createElement('img');
image.src = 'smile.png';
document.body.appendChild(img);

In this case, when the new <img> element has its src attribute set, the image is requested by the browser. Attaching it to the DOM will result in the image being displayed once it loads.

If we want to know when the image is loaded, we can use the load event:

var image = document.createElement('img');
image.addEventListener('load', function(event){
    console.log('loaded image');
});
image.src = 'smile.png';

Notice too that we add the event listener before we set the src attribute. If we did it the other way around, the resource may already be loaded before the listener takes effect - and it would never be invoked!

However, this approach can be cumbersome, and limits us to what resources can be bound to HTML elements. For more flexibility, we need to make the request directly, using AJAX. We’ll take a look at doing so after we cover HTTP in more depth.

Summary

In this chapter, we reviewed the Document Object Model (the DOM), the tree-like structure of HTML elements built by the browser as it parses an HTML document. We discussed how CSS rules are applied to nodes in this tree to determine how the final webpage will be rendered, and how JavaScript can be used to manipulate and transform the DOM (and the resulting webpage appearance).

We also discussed how JavaScript events work, and how this event-driven approach is the basis for implementing concurrency within the language. We’ll see this more as we delve into Node.js, which utilizes the same event-based concurrency model, in future chapters.

Finally, we discussed how supplemental files (images, videos, CSS files, JavaScript files) are loaded by the browser concurrently. We saw how this can affect the functioning of JavaScript that depends on certain parts of the page already having been loaded, and saw how we can use the load event to delay running scripts until these extra files have completed loading.

Introduction

At the heart of the world wide web is the Hyper-Text Transfer Protocol (HTTP). This is a protocol defining how HTTP servers (which host web pages) interact with HTTP clients (which display web pages).

It starts with a request initiated from the web browser (the client). This request is sent over the Internet using the TCP protocol to a web server. Once the web server receives the request, it must decide the appropriate response - ideally sending the requested resource back to the browser to be displayed. The following diagram displays this typical request-response pattern.

This HTTP request-response pattern is at the core of how all web applications communicate. Even those that use websockets begin with an HTTP request.

Browser Requests

Before we get too deep in the details of what a request is, and how it works, let’s explore the primary kind of request you’re already used to making - requests originating from a browser. Every time you use a browser to browse the Internet, you are creating a series of HTTP (or HTTPS) requests that travel across the networks between you and a web server, which responds to your requests.

To help illustrate how these requests are made, we’ll once again turn to our developer tools. Open the example page this link. On that tab, open your developer tools with CTRL + SHIFT + i or by right-clicking the page and selecting “Inspect” from the context menu. Then choose the “Network” tab:

The network tab displays details about each request the browser makes. Initially it will probably be empty, as it does not log requests while not open. Try refreshing the page - you should see it populate with information:

The first entry is the page itself - the HTML file. But then you should see entries for site.css, brazil.gif, fiber-4814456_960_720.jpg, jquery-3.5.1.slim.min.js, and site.js. Each of these entries represents an additional resource the browser fetched from the web in order to display the page.

Take, for example, the two images brazil.gif and fiber-4814456_960_720.jpg. These correspond to <img> tags in the HTML file:

<img alt="Paper scene from the film Brazil" src="brazil.gif"/>
<img alt="Fiber optic cables" src="https://cdn.pixabay.com/photo/2020/02/03/00/12/fiber-4814456_960_720.jpg"/>
        

The important takeaway here is that the image is requested separately from the HTML file. As the browser reads the page and encounters the <img> tag, it makes an additional request for the resource supplied in its src attribute. When that second request finishes, the downloaded image is added to the web page.

Notice too that while one image was on our webserver, the other is retrieved from Pixabay.com’s server.

Similarly, we have two JavaScript files:

<script src="https://code.jquery.com/jquery-3.5.1.slim.min.js" integrity="sha256-4+XzXVhsDmqanXGHaHvgh1gMQKX40OUvDEBTu8JcmNs=" crossorigin="anonymous"></script>
<script src="site.js"></script>

As with the images, one is hosted on our website, site.js, and one is hosted on another server, jquery.com.

Both the Pixabay image and the jQuery library are hosted by a Content Delivery Network- a network of proxy servers that are distributed geographically in such a way that a request for a resource they hold can be processed from a nearby server. Remember that the theoretical maximum speed for internet transmissions is the speed of light (for fiber optics) or electrons in copper wiring. Communication is further slowed at each network switch encountered. Serving files from a nearby server can prove very efficient at speeding up page loads because of the shorter distance and smaller number of switches involved.

A second benefit of using a CDN to request the JQuery library is that if the browser has previously downloaded the library when visiting another site it will have cached it. Using the cached version instead of making a new request is much faster. Your app will benefit by faster page loads that use less bandwidth.

Notice too that the jQuery <script> element also uses the integrity attribute to allow the browser to determine if the library downloaded was tampered with by comparing cryptographic tokens. This is an application of Subresource Integrity, a feature that helps protect your users. As JavaScript can transform the DOM, there are incentives for malicious agents to supplant real libraries with fakes that abuse this power. As a web developer you should be aware of this, and use all the tools at your disposal to keep your users safe.

You can use the network tab to help debug issues with resources. Click on one of the requested resources, and it will open up details about the request:

Notice that it reports the status code along with details about the request and response, and provides a preview of the requested resource. We’ll cover what these all are over the remainder of this chapter. As you learn about each topic, you may want to revisit the tab with the example to see how these details correspond to what you are learning.

Request Format

So now that we’ve seen HTTP Requests in action, let’s examine what they are. A HTTP Request is just a stream of text that follows a specific format and sent from a client to a server.

It consists of one or more lines terminated by a CRLF (a carriage return and a line feed character, typically written \r\n in most programming languages).

1. A request-line describing the request
2. Additional optional lines containing HTTP headers. These specify details of the request or describe the body of the request
3. A blank line, which indicates the end of the request headers
4. An optional body, containing any data belonging of the request, like a file upload or form submission. The exact nature of the body is described by the headers.

The Request-Line

The request-line follows the format

Request-Line = Method SP Request-URI SP HTTP-Version CRLF

The Method refers to the HTTP Request Method (often GET or POST).

SP refers to the space character.

The Request-URI is a Universal Request Indicator, and is typically a URL or can be the asterisk character (*), which refers to the server instead of a specific resource.

HTTP-Version refers to the version of the HTTP protocol that will be used for the request. Currently three versions of the protocol exist: HTTP/1.0, HTTP/1.1, and HTTP/2.0. Most websites currently use HTTP/1.1 (HTTP/2.0 introduces many changes to make HTTP more efficient, at the cost of human readability. Currently it is primarily used by high-traffic websites).

Finally, the CRLF indicates a carriage return followed by a line feed.

For example, if we were requesting the about.html page of a server, the request-line string would be:

GET /about.html HTTP/1.1\r\n

The Headers

Header lines consist of key-value pairs, typically in the form

Header = Key: Value CRLF

Headers provide details about the request, for example, if we wanted to specify we can handle the about.html page data compressed with the gzip algorithm, we would add the header:

Accept-Encoding: compress, gzip\r\n

The server would then know it could send us a zipped version of the file, resulting in less data being sent from the server to the client.

If our request includes a body (often form data or a file upload), we need to specify what that upload data is with a Content-Type header and its size with a Content-Length header, i.e.:

Content-Length: 26012 Content-Type: image/gif

A Blank Line

The header section is followed by a blank line (a CRLF with no characters before it). This helps separate the request metadata from the request body.

The Request Body

The body of the request can be text (as is the case for most forms) or binary data (as in an image upload). This is why the Content-Type header is so important for requests with a body; it lets the server know how to process the data. Similarly, the Content-Length header lets us know how many bytes to expect the body to consist of.

It is also acceptable to have no body - this is commonly the case with a GET request. If there is no body, then there are also no required headers. A simple get request can therefore consist of only the request-line and blank line, i.e.:

GET /about.html HTTP/1.1\r\n\r\n

Request Methods

The first line of the HTTP request includes the request method, which indicates what kind of action the request is making of the web server (these methods are also known as HTTP Verbs). The two most common are GET and POST, as these are supported by most browsers.

Commonly Used HTTP Methods

The following requests are those most commonly used in web development. As noted before GET and POST requests are the most commonly used by web browsers, while GET, PUT, PATCH, and DELETE are used by RESTful APIs. Finally, HEAD can be used to optimize web communications by minimizing unnecessary data transfers.

GET

A GET request seeks to retrieve a specific resource from the web server - often an HTML document or binary file. GET requests typically have no body and are simply used to retrieve data. If the request is successful, the response will typically provide the requested resource in its body.

HEAD

A HEAD request is similar to a GET request, except the response is not expected to provide a body. This can be used to verify the type of content of the resource, the size of the resource, or other metadata provided in the response header, without downloading the full data of the resource.

POST

The POST request submits an entity to the resource, i.e. uploading a file or form data. It typically will have a body, which is the upload or form.

PUT

The PUT request is similar to a POST request, in that it submits an entity as its body. It has a more strict semantic meaning though; a PUT request is intended to replace the specified resource in its entirety.

PATCH

The PATCH request is also similar to POST and PUT requests - it submits an entity as its body. However, its semantic meaning is to only apply partial modifications to the specified entity.

DELETE

As you might expect, the DELETE method is used to delete the specified resource from the server.

URIs and URLs

Before a web request can be made, the browser needs to know where the resource requested can be found. This is the role that a Universal Resource Locator (a URL) plays. A URL is a specific kind of Universal Resource Indicator (URI) that specifies how a specific resource can be retrieved.

A URI consists of several parts, according to the definition (elements in brackets are optional):

URI = scheme:[//[userinfo@]host[:port]]path[?query][#fragment]

Let’s break this down into individual parts:

scheme: The scheme refers to the resource is identified and (potentially) accessed. For web development, the primary schemes we deal with are http (hyper-text transfer protocol), https (secure hyper-text transfer protocol), and file (indicating a file opened from the local computer).

userinfo: The userinfo is used to identify a specific user. It consists of a username optionally followed by a colon (:) and password. We will discuss its use in the section on HTTP authentication, but note that this approach is rarely used today, and carries potential security risks.

host: The host consists of either a fully quantified domain name (i.e. google.com, cs.ksu.edu, or gdc.ksu.edu) or an ip address (i.e. 172.217.1.142 or [2607:f8b0:4004:803::200e]). IPv4 addresses must be separated by periods, and IPv6 addresses must be closed in brackets. Additionally, web developers will often use the loopback host, localhost, which refers to the local machine rather than a location on the network.

port: The port refers to the port number on the host machine. If it is not specified (which is typical), the default port for the scheme is assumed: port 80 for HTTP, and port 443 for HTTPS.

path: The path refers to the path to the desired resource on the server. It consists of segments separated by forward slashes (/).

query: The query consists of optional collection of key-value pairs (expressed as key:value), separated by ampersands (&), and preceded by a question mark (?). The query string is used to supply modifiers to the requested resource (for example, applying a filter or searching for a term).

fragment: The fragment is an optional string proceeded by a hashtag (#). It identifies a portion of the resource to retrieve. It is most often used to auto-scroll to a section of an HTML document, and also for navigation in some single-page web applications.

Thus, the URL https://google.com indicates we want to use the secure HTTP scheme to access the server at google.com using its port 443. This should retrieve Google’s main page.

Similarly, the url https://google.com/search?q=HTML will open a Google search result page for the term HTML (Google uses the key q to identify search terms).

Request Headers

Request headers take the form of key-value pairs, separated by colons : and terminated with a CRLF (a carriage return and line feed character). For example:

Accept-Encoding: gzip

Indicates that the browser knows how to accept content compressed in the Gzip format.

Note that request headers are a subset of message headers that apply specifically to requests. There are also message headers that apply only to HTTP responses, and some that apply to both.

As HTTP is intended as an extensible protocol, there are a lot of potential headers. IANA maintains the official list of message headers as well as a list of proposed message headers. You can also find a categorized list in the MDN Documentation

While there are many possible request headers, some of the more commonly used are:

Accept Specifies the types a server can send back, its value is a MIME type.

Accept-Charset Specifies the character set a browser understands.

Accept-Encoding Informs the server about encoding algorithms the client can process (most typically compression types)

Accept-Language Hints to the server what language content should be sent in.

Authorization Supplies credentials to authenticate the user to the server. Will be covered in the authentication chapter.

Content-Length The length of the request body sent, in octets

Content-Type The MIME type of the request body

Content-Encoding The encoding method of the request body

Cookie Sends a site cookie - see the section on cookies later

User-Agent A string identifying the agent making the request (typically a browser name and version)

Request Body

After the request headers and an extra CRLF (carriage return and line feed) is the request body.

For GET and DELETE requests, there is no body. For POST, PUT, and PATCH, however, this section should contain the data being sent to the server. If there is a body, the headers should include Content-Type and Content-Length. The Content-Length is always provided as a count of octets (a set of eight bits). Thus, binary data is sent as an octet stream. Text data is typically sent in UTF-8 encoding.

Two body formats bear special mention: application/x-www-form-urlencoded and multipart/form-data. These encodings are commonly used for submitting HTML forms, and will be covered in more detail in the Form Data chapter.

Response Format

Similar to an HTTP Request, an HTTP response is typically a stream of text and possibly data:

It consists of one or more lines of text, terminated by a CRLF (sequential carriage return and line feed characters):

1. A status-line indicating the HTTP protocol, the status code, and a textual status
2. Optional lines containing the Response Headers. These specify the details of the response or describe the response body
3. A blank line, indicating the end of the response metadata
4. An optional response body. This will typically be the text of an HTML file, or binary data for an image or other file type, or a block of bytes for streaming data.

The Status-Line

The status-line follows the format

Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF

The HTTP-Version indicates the version of the HTTP protocol that is being used (HTTP/1.0, HTTP/1.1, or HTTP/2.0).

SP refers to a space character.

The Status-Code is a three-digit numeric representation of the response status. Common codes include 200 (OK), 404 (Not Found), and 500 (Server Error).

The Reason-Phrase is a plain-text explanation of the status code.

Response Headers

Just like HTTP Requests, a HTTP response can contain headers describing the response. If the response has a body, a Content-Type and Content-Length header would be expected.

A Blank Line

The header section is followed by a blank line (a CRLF with no characters before it). This helps separate the response metadata from the response body.

Response Body

The response body contains the data of the response. it might be text (as is typically the case with HTML, CSS, or JavaScript), or a binary file (an image, video, or executable). Additionally, it might only be a sequence of bytes, as is the case for streaming media.

Status Codes

The status-line consists of a numeric code, followed by a space, and then a human-readable status message that goes with the code. The codes themselves are 3-digit numbers, with the first number indicating a general category the response status falls into. Essentially, the status code indicates that the request is being fulfilled, or the reason it cannot be.

1XX Status Codes

Codes falling in the 100’s provide some kind of information, often in response to a HEAD or upgrade request. See the MDN Documentation for a full list.

2XX Status Codes

Codes in the 200’s indicate success in some form. These include:

200 OK A status of 200 indicates the request was successful. This is by far the most common response.

201 Created Issued in response to a successful POST request, indicates the resource POSTed to the server has been created.

202 Accepted Indicates the request was received but not yet acted upon. This is used for batch-style processes. An example you may be familiar with is submitting a DARS report request - the DARS server, upon receiving one, adds it to a list of reports to process and then sends a 202 response indicating it was added to the list, and should be available at some future point.

There are additional 200 status codes. See the MDN Documentation for a full list.

3XX Status Codes

Codes in the 300’s indicate redirects. These should be used in conjunction with a Location response header to notify the user-agent where to redirect. The three most common are:

301 Moved Permanently Indicates the requested resource is now permanently available at a different URI. The new URI should be provided in the response, and the user-agent may want to update bookmarks and caches.

302 Found Also redirects the user to a different URI, but this redirect should be considered temporary and the original URI used for further requests.

304 Not Modified Indicates the requested resource has not changed, and therefore the user-agent can use its cached version. By sending a 304, the server does not need to send a potentially large resource and consume unnecessary bandwidth.

There are additional 300 status codes. See the MDN Documentation for a full list.

4XX Status Codes

Codes in the 400’s indicate client errors. These center around badly formatted requests and authentication status.

400 Bad Request is a request that is poorly formatted and cannot be understood.

401 Unauthorized means the user has not been authenticated, and needs to log in.

403 Forbidden means the user does not have permissions to access the requested resource.

404 Not Found means the requested resource is not found on the server.

There are many additional 400 status codes. See the MDN Documentation for a full list.

5XX Status Codes

Status codes in the 500’s indicate server errors.

500 Server Error is a generic code for “something went wrong in the server.”

501 Not Implemented indicates the server does not know how to handle the request method.

503 Service Unavailable indicates the server is not able to handle the request at the moment due to being down, overloaded, or some other temporary condition.

There are additional 500 status codes. See the MDN Documentation for a full list.

Response Headers

Response headers take the form of key-value pairs, separated by colons : and terminated with a CRLF (a carriage return and line feed character), just like Request Headers (both are types of Message Headers). For example, this header:

Expires: Wed, 12 Jun 2019 08:00:00 CST

indicates to the browser that this content will expire June 12, 2019 at 8AM Central Standard Time. The browser can use this value when populating its cache, allowing it to use the cached version until the expiration time, reducing the need to make requests.

Note that response headers are a subset of message headers that apply specifically to requests. As we’ve seen there are also message headers that apply only to HTTP requests, and some that apply to both.

As HTTP is intended as an extensible protocol, there are a lot of potential headers. IANA maintains the official list of message headers as well as a list of proposed message headers. You can also find a categorized list in the MDN Documentation

While there are many possible response headers, some of the more commonly used are:

Allow Lists the HTTP Methods that can be used with the server

Content-Length The length of the response body sent, in octets

Content-Type The MIME type of the response body

Content-Encoding The encoding method of the response body

Location Used in conjunction with redirects (a 301 or 302 status code) to indicate where the user-agent should be redirected to.

Server Contains information about the server handling the request.

Set-Cookie Sets a cookie for this server on the client. The client will send back the cookie on subsequent requests using the Cookie header.

We’ll make use of these headers as we start writing web servers.

Response Body

After the response headers and an extra CRLF (carriage return and line feed) is the response body.

The body is typically text (for HTML, CSS, JavaScript, and other text files) or binary data (for images, video, and other file types).

Setting the Content-Type and Content-Length headers lets the web client know what kind of data, and how much of it, should be expected. If these headers are not supplied in the response, the browser may treat the body as a blob of binary data, and only offer to save it.

HTTP 2.0

You may have noticed that the earlier parts of this chapter focused on HTTP version 1.1, while mentioning version 2.0. You might wonder why we didn’t instead look at version 2.0 - and it’s a valid question.

In short, HTTP 2.0 was created to make the request-response pattern of the web more efficient. One method it uses to do so is switching from text-based representations of requests and responses to binary-based representations. As you probably remember from working with File I/O, binary files are much smaller than the equivalent text file. The same is true of HTTP requests and responses. But the structure of HTTP 2.0 Requests and Responses are identical to HTTP 1.1 - they are simply binary (an hence, harder to read).

But this is not the only improvement in HTTP 2.0. Consider the request-response pattern we discussed earlier in the chapter:

To display a webpage, the browser must first request the page’s HTML data. As it processes the returned HTML, it will likely encounter HTML <img>, <link>, and <src> tags that refer to other resources on the server. To get these resources, it will need to make additional requests. For a modern webpage, this can add up quickly! Consider a page with 20 images, 3 CSS files, and 2 JavaScript files. That’s 25 separate requests!

One of the big improvements between HTTP/1.0 and HTTP/1.1 was that HTTP/1.1 does not close its connection to the server immediately - it leaves a channel open for a few seconds. This allows it to request these additional files without needing to re-open the connection between the browser and the server.

HTTP/2.0 takes this thought a step farther, by trying to anticipate the browser’s additional requests. In the HTTP/2.0 protocol, when a browser requests an HTML page, the server can push the additional, linked files as part of the response. Thus, the entire page content can be retrieved with a single request instead of multiple, separate requests. This minimizes network traffic, allowing the server to handle more requests, and speeds up the process of rendering pages on a client’s machine.

Statelessness and Scaling

One final point to make about Hyper-Text Transfer Protocol. It is a stateless protocol. What this means is that the server does not need to keep track of previous requests - each request is treated as though it was the first time a request has been made.

This approach is important for several reasons. One, if a server must keep track of previous requests, the amount of memory required would grow quickly, especially for popular web sites. We could very quickly grow past the memory capacity of the server hardware, causing the webserver to crash.

A second important reason is how we scale web applications to handle more visitors. We can do vertical scaling - increasing the power of the server hardware - or horizontal scaling - adding additional servers to handle incoming requests. Not surprisingly, the second option is the most cost-effective. But for it to work, requests need to be handed quickly to the first available server - there is no room for making sure a subsequent request is routed to the same server.

Thus, HTTP was designed as a stateless protocol, in that each new HTTP request is treated as being completely independent from all previous requests from the same client. This means that when using horizontal scaling, if the first request from client BOB is processed by server NANCY, and the second request from BOB is processed by server MARGE, there is no need for MARGE to know how NANCY responded. This stateless property was critical to making the world-wide-web even possible with the earliest internet technologies.

Of course, you probably have experience with websites that do seem to keep track of state - i.e. online stores, Canvas, etc. One of the challenges of the modern web was building state on top of a stateless protocol; we’ll discuss the strategies used to do so in later chapters.

Summary

In this chapter, we explored the Hyper-Text Transfer Protocol (HTTP), the technology used to exchange data across the web. We saw how requests and responses are simply well-structured streams of text or data exchanged across a packet-switched network. We examined the structure of these requests, and saw how they all use a similar pattern of a request or response line, a series of headers carrying metadata about the request or response, and an optional body.

We also discussed HTTP request methods like GET and POST, and HTTP response codes like a 404 and what these terms mean. We explored how HTTP has evolved from HTTP 1.0 to 1.1 to 2.0. And we discussed how HTTP is a stateless protocol, and how that statelessness is key to making websites that can serve thousands if not millions of users.

We’ll continue to reference the content of this chapter as we move forward, as HTTP is one of the core web technologies along with HTML, CSS, and JavaScript.

Introduction

JavaScript makes extensive use of asynchronous processing to tackle the challenge of concurrency. This includes the events we’ve already talked about (user events, network events and timers), but it has also been expanded to provide even more powerful features. The XMLHTTPRequest object allows JavaScript to request additional resources directly in an asynchronous manner, and the more recent Fetch API updates that approach. Web workers allow parallel JavaScript processes to be run in the browser. And new ES6 syntax and constructs like Promises and the async/await keywords make asynchronous functions easier to reason about and use. In this chapter, we will explore each of these topics.

Concurrency Approaches

Concurrency means “doing more than one thing at the same time.” In computer science, concurrency can refer to (1) structuring a program or algorithm so that it can be executed out-of-order or in partial order, or (2) actually executing computations in parallel. In modern-day programming, we’re often talking about both. But to help us develop a stronger understanding, let’s look at the two ideas one-at-a-time, and then bring it together.

Concurrency

One of the earliest concurrency problems computer science dealt with was the efficient use of early computers. Consider a mainframe computer like the PDP-1, once a staple of the university computer science department. In 1963, the base price of a PDP-1 was $120,000. Adjusted for inflation, this would be a price of a bit more than one million dollars in 2020! That’s a pretty big investment! Any institution, be it a university, a corporation, or a government agency that spent that kind of money would want to use their new computer as efficiently as possible.

Consider when you are working on a math assignment and using a calculator. You probably read your problem carefully, write out an equation on paper, and then type a few calculations into your calculator, and copy the results to your paper. You might write a few lines as you progress through solving the problem, then punch a new calculation into your calculator. Between computations, your calculator is sitting idle - not doing anything. Mainframe computers worked much the same way - you loaded a program, it ran, and spat out results. Until you loaded a new program, the mainframe would be idle.

Batch Processing

An early solution was the use of batch processing, where programs were prepared ahead of time on punch-card machines or the like, and turned over to an IT department team that would then feed these programs into the computer. In this way, the IT staff could keep the computer working as long as there was batched work to do. While this approach kept the computer busy, it was not ideal for the programmers. Consider the calculator example - it would be as if you had to write out your calculations and give them to another person to enter into the calculator. And they might not get you your results for days!

Can you imagine trying to write a program that way? In the early days that was exactly how CS students wrote programs - they would write an entire program on punch cards, turn it in to the computer staff to be batched, and get the results once it had been run. If they made a mistake, it would require another full round of typing cards, turning them in, and waiting for results!

Batch processing is still used for some kinds of systems - such as the generation of your DARS report at K-State, for sending email campaigns, and for running jobs on Beocat and other supercomputers. However, in mainframe operations it quickly was displaced by time sharing.

Time Sharing

Time-sharing is an approach that has much in common with its real-estate equivalent that shares its name. In real estate, a time-share is a vacation property that is owned by multiple people, who take turns using it. In a mainframe computer system, a time sharing approach likewise means that multiple people share a single computer. In this approach, terminals (a monitor and keyboard) are hooked up to the mainframe. But there is one important difference between time-sharing real estate and computers, which is why we can call this approach concurrent.

Let’s return to the example of the calculator. In the moments between your typing an expression and reading the results, another student could type in their expression, and get their results. A time-sharing mainframe does exactly that - it take a few fractions of a second to advance each users’ program, switching between different users at lightning speed. Consider a newspaper where twenty people might we writing stories at a given moment - each terminal would capture key presses, and send them to the mainframe when it gave its attention, which would update the text editor, and send the resulting screen data back to the terminal. To the individual users, it would appear the computer was only working with them. But in actuality it was updating all twenty text editor program instances in real-time (at least to human perception).

Like batch processing, time-sharing is still used in computing today. If you’ve used the thin clients in the DUE 1114 lab, these are the current-day equivalents of those early terminals. They’re basically a video card, monitor, and input device that are hooked up to a server that runs multiple VMs (virtual machines), one for each client, and switches between them constantly updating each.

Multitasking

The microcomputer revolution did not do away with concept. Rather, modern operating systems still use the basic concept of the approach, though in the context of a single computer it is known as multitasking. When you write a paper now, your operating system is switching between processes in much the same way that time-sharing switched between users. It will switch to your text editor, processing your last keystroke and updating the text on screen. Then it will shift to your music player and stream the next few thousand bytes of the song you’re listening to the sound card. Then it will switch to your email program which checks the email server and it will start to notify you that a new email has come in. Then it will switch back to your text editor.

The thin clients in DUE 1114 (as well as the remote desktops) are therefore both time-sharing between VMs and multitasking within VMs.

Parallel Processing

The second approach to concurrency involves using multiple computers in parallel. K-State’s Beocat is a good example of this - a supercomputer built of a lot of individual computers. But your laptop or desktop likely is as well; if you have a multi-core CPU, you actually have multiple processors built into your CPU, and each can run separate computational processes. This, it is entirely possible that as you are writing your term paper the text editor is running on one processor, your email application is using a second one, and your music is running on a third.

In fact, modern operating systems use both multitasking and parallel processing in tandem, spreading out the work to do across however many cores are available, and swapping between active processes to on those cores. Some programs also organize their own computation to run on multiple processors - your text editor might actually be handling your input on one core, running a spellcheck on a second, and a grammar check on a third.

Remember our earlier discussion about scaling web servers? This is also a parallel processing approach. Incoming HTTP requests are directed by a load balancer to a less-busy server, and that server formulates the response.

Multithreading

Individual programs can also be written to execute on multiple cores. We typically call this approach Multithreading, and the individually executing portions of the program code threads.

These aren’t the only ways to approach concurrency, but they are ones we commonly see in practice. Before we turn our attention to how asynchronous processes fit in though, we’ll want to discuss some of the challenges that concurrency brings.

Concurrency Challenges

Implementing concurrency in computing systems comes with some specific challenges. Consider the multitasking approach where we have your text editor and your music player running at the same time. As the text editor process yields to the music player, the data and program elements it had loaded up into working memory, needs to be cleared out and replaced with the music player’s data and program. However, the music player’s data and program need to be retained somewhere so that they can be swapped back in when the music player yields.

Modern operating systems handle this challenge by dividing working memory amongst all programs running (which is why the more programs you run, the more RAM you consume). Each running process is assigned a block of memory and only allowed to use that memory. Moreover, data copied into the CPU (the L2 cache, L1 cache, and registers) may also be cached in memory for later restoration. You’ll learn more about the details of this process if you take an Operating Systems course. But it is very important that each running program is only allowed to make changes in its own assigned memory space. If it wasn’t, it could overwrite the data of another task!

While operating systems normally manage the challenges of concurrency between running programs, when the program itself is written to be concurrent, the program itself must be built to avoid accidentally overwriting its own memory in unexpected ways. Consider a text editor - it might have its main thread handling user input, and a second thread handling spell checking. Now the user has typed “A quick brow”, and the spell checker is finding that “brow” is misspelled. It might try to underline the line in red, but in the intervening time, the user has deleted “brow”, so the underlining is no longer valid!

Or consider image data. Applying a filter to an image is a computationally costly operation - typically requiring visiting each pixel in the image, and for some filters, visiting each pixel around each pixel as part of the transformation. This would be a good use-case for multi-threading. But now imagine two filters working in parallel. One might be applying a blur, and the other a grayscale transformation. If they were overwriting the old image data with their new filtered version, and were working at the same time, they might both start from the original pixels in the upper-right-hand corner. The blur would take longer, as it needs to visit neighboring pixels. So the grayscale filter writes out the first hundred pixels, and then the blur writes out its first hundred, over-writing the grayed pixels with blurred color pixels. Eventually, the grayscale filter will get so far ahead of the blur filter that the blur filter will be reading in now-greyed out pixels, and blurring them. The result could well be a mishmash of blurred color and gray-and-white splotches.

There are many different approaches that can be used to manage this challenge. One is the use of locks - locking a section of memory so only one thread can access it while it makes changes. In the filter example, the grayscale filter could lock the image data, forcing the blur filter to wait until it finishes. Locks work well, but must be carefully designed to avoid race conditions - where two threads cannot move forward because the other thread has already locked a resource the blocked thread needs to finish its work.

Asynchronous programming is another potential solution, which we’ll look at next.

Asynchronous Programming

In asynchronous programming, memory collisions are avoided by not sharing memory between threads. A unit of work that can be done in parallel is split off and handed to another thread, and any data it needs is copied into that threads’ memory space. When the work is complete, the second thread notifies the primary thread if the work was completed successfully or not, and provides the resulting data or error.

In JavaScript, this notification is pushed into the event queue, and the main thread processes it when the event loop pulls the result off the event queue. Thus, the only memory that is shared between the code you’ve written in the Event Loop and the code running in the asynchronous process is the memory invovled in the event queue. Keeping this memory thread-safe is managed by the JavaScript interpreter. Thus, the code you write (which runs in the Event Loop) is essentially single-threaded, even if your JavaScript application is using some form of parallel processing!

Let’s reconsider a topic we’ve already discussed with this new understanding - timers. When we invoke setTimer(), we are creating a timer that is managed asynchronously. When the timer elapses, it creates a timer ’event’ and adds it to the event queue. We can see this in the diagram below.

However, the timer is not actually an event, in the same sense that a 'click' or 'keydown' event is… in that those events are provided to the browser from the operating system, and the browser passes them along into the JavaScript interpreter, possibly with some transformation. In contrast, the timer is created from within the JavaScript code, though its triggering is managed asynchronously.

In fact, both timers and events represent this style of asynchronous processing - both are managed by creating messages that are placed on the event queue to be processed by the interpreter. But the timer provides an important example of how asynchronous programming works. Consider this code that creates a timer that triggers after 0 milliseconds:

setTimeout(()=>{
    console.log("Timer expired!");
}, 0);
console.log("I just set a timer.");

What will be printed first? The “Timer expired!” message or the “I just set a timer.” message?

See for yourself - try running this code in the console (you can click the “console” tab below to open it).

The answer is that “I just set a timer” will always be printed first, because the second message won’t be printed until the event loop pulls the timer message off the queue, and the line printing “I just set a timer” is executed as part of this pass in the event loop. The setTimeout() and setInterval() functions are what we call asynchronous functions, as they trigger an asynchronous process. Once that process is triggered, execution immediately continues within the event loop, while the triggered process runs in parallel. Asynchronous functions typically take a function as an argument, known as a callback function, which will be triggered when the message corresponding to the asynchronous process is pulled off the event queue.

As JavaScript was expanded to take on new functionality, this asynchronous mechanism was re-used. Next, we’ll take a look at an example of this in the use of web workers.

Web Workers

As JavaScript began to be used to add more and more functionality to web applications, an important limitation began to appear. When the JavaScript interpreter is working on a big task, it stays in the event loop a long time, and does not pull events from the event queue. The result is the browser stops responding to user events… and seems to be frozen. On the other hand - some programs will never end. Consider this one:

while(true) {
    console.log("thinking...");
}

This loop has no possible exit condition, so if you ran it in the browser, it would run infinitely long… and the page would never respond to user input, because you’d never pull any events of the event queue. One of the important discoveries in computer science, the Halting Problem tackles exactly this issue - and Alan Turing’s proof shows that a program to determine if another program will halt for all possible programs cannot be written.

Thus, browsers instead post warning messages after execution has run for a significant amount of time, like this one:

So, if we want to do a long-running computation, and not have the browser freeze up, we needed to be able to run it separately from the thread our event loop is running on. The web worker provides just this functionality.

A web worker is essentially another JavaScript interpreter, running a script separate from the main thread. As an interpreter, it has its own event loop and its own memory space. Workers and the main thread can communicate by passing messages, which are copied onto their respective event queues. Thus, communication between the threads is asynchronous.

An Example

You can see an example of such a web worker by using this link to open another tab in your browser. This example simulates a long-running process of n seconds either in the browser’s main thread or in a web worker. On the page is also three colored squares that when clicked, shift colors. Try changing the colors of the squares while simulating a several-second process in both ways. See how running the process on the main thread freezes the user interface?

Using Web Workers

Web workers are actually very easy to use. A web worker is created by constructing a Worker object. It takes a single argument - the JavaScript file it will execute (which should be hosted on the same server). In the example, this is done with:

// Set up the web worker
var worker = new Worker('stall.js');

The stall.js file contains the script the worker will execute - we’ll take a look at it in a minute.

Once created, you can attach a listener to the Worker. It has two events:

• message - a deserialized message sent from the web worker
• mesageerror - a message sent from the web worker that was not serializable

You can use worker.addEventListener() to add these, or you can assign your event listener to the event handler properties. Those properties are:

• Worker.onmessage - triggered when the message event happens
• Worker.onmessageerror - triggered when the messageerror event happens

Additionally, there is an error handler property:

• Worker.onerror

Which triggers when an uncaught error occurs on the web worker.

In our example, we listen for messages using the Worker.onmessage property:

// Set up message listener
worker.onmessage = function(event){
    // Signal completion
    document.querySelector('#calculation-message').textContent = `Calculation complete!`;
}

If the message was successfully deserialized, it’s data property contains the content of the message, which can be any valid JavaScript value (an int, string, array, object, etc). This gives us a great deal of flexibility. If you need to send more than one type of message, a common strategy is to send a JavaScript object with a type property, and additional properties as needed, i.e.:

var messageData1 = {
    type: "greeting",
    body: "Hello!  It's good to see you."
}

var messageData2 = {
    type: "set-color",
    color: "#ffaacc"
}

We can send messages to the web worker with Worker.postMessage(). Again, these messages can be any valid JavaScript value:

worker.postMessage("Foo");

worker.postMessage(5);

worker.postMessage({type: "greetings", body: "Take me to your leader!"});

In our example, we send the number of seconds to wait as an integer parsed from the <input> element:

// Get the number to calculate the Fibonacci number for and convert it from a string to a base 10 integer
var n = parseInt(document.querySelector('#n').value, 10);

// Stall for the specified amount of time
worker.postMessage(n);

Whatever data we send as the message is copied into the web worker’s memory using the structured clone algorithm. We can also optionally transfer objects to the web worker instead of copying them by providing them as a second argument. This transfers the memory holding them from one thread to the other. This makes them unavailable on the original thread, but is much faster than copying when the object is large. It is used for sending objects like ArrayBuffer, MessagePort, or ImageBitmap. Transferred objects also need to have a reference in the first argument.

The Web Worker Context

For the JavaScript executing in the web worker, the context is a bit different. First, there is no document object model, as the web worker cannot make changes to the user interface. Likewise there is no global window object. However, many of the normal global functions and properties not related to the user interface are available, see functions and classes available to web workers for details.

The web worker has its own unique global scope, so any variables declared in your main thread won’t exist here. Likewise, varibles declared in the worker will not exist in the main scope either. The global scope of the worker has mirror events and properties to the Worker - we can listen for messages from the main thread using the onmessage and onmessageerror properties, and send messages back to the main thread with postMessage().

The complete web worker script from the example is:

/** @function stall 
 * Synchronously stalls for the specified amount of time 
 * to simulate a long-running calculation
 * @param {int} seconds - the number of seconds to stall
 */
function stall(seconds) {
    var startTime = Date.now();
    var endTime = seconds * 1000 + startTime;
    while(true) {
        if(Date.now() > endTime) break;
    }
}


/**
 * Message handler for messages from the main thread
 */
onmessage = function(event) {
    // stall for the specified amount of time 
    stall(event.data);
    // Send an answer back to the main thread
    postMessage(`Stalled for ${event.data} seconds`);
    // Close the worker since we create a
    // new worker with each stall request.
    // Alternatively, we could re-use the same
    // worker.
    close();
};

Workers can also send AJAX requests, and spawn additional web workers! In the case of spawning additional workers, the web worker that creates the child worker is treated as the main thread.

Other kinds of Web Workers

The web workers we’ve discussed up to this point are basic dedicated workers. There are also several other kinds of specialized web workers:

• Shared workers are shared between several scripts, possibly even running in different <iframe> elements. These are more complex than a dedicated worker and communicate via ports. See MDN’s SharedWorker article for information.
• Service workers act as proxy servers between the server and the web app, for the purpose of allowing web apps to be used offline. We’ll discuss these later in the semester, but you can read up on them in hte mdn ServiceWorker article.
• Audio workers allow for direct scripting of audio processing within a web worker context. See the mdn AudioWorker article for details.

Callbacks

JavaScript implements its asynchronous nature through callbacks - functions that are invoked when an asynchronous process completes. We see this in our discussion of timers like setTimeout() and with our web workers with the onmessage event handler. These demonstrate two possible ways of setting a callback. With setTimeout() we pass the callback as a function parameter, i.e.:

function timeElapsed() {
    console.log("Time has elapsed!");
}
// Set a timer for 1 second, and trigger timeElapsed() when the timer expires
setTimeout(timeElapsed, 1000);

With webworkers, we assign a function to a property of the worker (the onmessage variable):

function messageReceived(message) {
    console.log("Received " + message);
}
// Set the event listener
onmessage = messageReceived;

Remember, a callback is a function, and in JavaScript, functions are first-order: we can assign them as a variable or pass them as an argument to a function! We can also define a callback asynchronously as part of the argument, as we do here:

setTimeout(function() {
    console.log("time elapsed")
}, 1000);

Or using lambda syntax:

setTimeout(() => {
    console.log("time elapsed")
}, 1000);

These are roughly equivalent to passing timeElapsed() in the first example - and you’ll likely see all three approaches when you read others’ code.

Callback Hell

Callbacks are a powerful mechanism for expressing asynchronicity, but overuse can lead to difficult to read code - a situation JavaScript programmers refer to as “callback hell”. This problem became especially pronounced once programmers began using Node to build server-side code, as Node adopted the event-based callback asynchronous model of JavaScript for interactions with the file system,databases, etc. (we’ll cover Node in the next chapter).

Consider this example, which logs a user into a website:

webapp.get('/login', (req, res) => {
    parseFormData(req, res, (form) => {
        var username = form.username;
        var password = form.password;
        findUserInDatabase(username, (user) => {
            encryptPassword(password, user.salt, (hash) => {
                if(hash === user.passwordHash) 
                    res.setCookie({user: user});
                    res.end(200, "Logged in successfully!");
                else
                    res.end(403, "Unknown username/password combo");
            });
        });
    });
});

Don’t work about the exact details of the code, but count the number of nested callbacks. There are four! And reasoning about this deeply nested code starts getting pretty challenging even for an experienced programmer. Also, handling errors in this nested structure requires thinking through the nested structure.

There are strategies to mitigate this challenge in writing your code, including:

1. Keeping your code shallow
2. Modularizing your code
3. Handling every single error.

The site callbackhell.com offers a detailed discussion of these strategies.

As JavaScript matured, additional options were developed for handling this complexity - Promises and the async and await keywords. We’ll talk about them next.

Promises

Promises replace the callback mechanism with a JavaScript object, a Promise. In many ways, this is similar to the XMLHttpRequest object that is at the heart of AJAX. You can think of it as a state machine that is in one of three states: pending, fulfilled, or rejected.

A promise can be created by wrapping an asynchronous call within a new Promise object. For example, we can turn a setTimeout() into a promise with:

var threeSecondPromise = new Promise((resolve, reject) => {
    setTimeout(() => {
        resolve("Timer elapsed");
    }, 300);
});

We can also create a promise that immediately resolves using Promise.resolve(), i.e.:

var fifteenPromise = Promise.resolve(15);

This promise is never in the pending state - it starts as resolved. Similarly, you can create a promise that starts in the rejected state with Promise.reject():

var failedPromise = Promise.reject("I am a failure...");

You can also pass an error object to Promise.reject().

Using Promise.prototype.then()

What makes promises especially useful is their then() method. This method is invoked when the promise finishes, and is passed whatever the promise resolved to, i.e. the string "Timer elapsed" in the example above. Say we want to log that result to the console:

threeSecondPromise.then(result => {console.log(result)});

This is a rather trivial example, but we can use the same approach to define a new method for creating timers that might seem more comfortable for object-oriented programmers:

function createTimer(milliseconds) {
    return new Promise((resolve, reject) => {
        setTimeout(() => {
            resolve();
        }, milliseconds);
    });
}

With this method, we can create a timer to do any arbitrary action, i.e.:

// Say "Hello Delayed World" after five seconds
createTimer(5000).then(() => console.log("Hello delayed World!"));

Using Promise.prototype.catch()

In addition to the then() method, promises also provide a catch() method. This method handles any errors that were thrown by the promise. Consider this function that returns a promise to compute an average:

function computeAverage(numbers)
{
    return new Promise((resolve, reject) => {
        // Sum the numbers
        var sum = numbers.reduce((acc, value) => acc + value);
        // Compute the average 
        var average = sum / numbers.length;
        resolve(average);
    });
}

Try copying this code into the console, and then run some examples, i.e.:

computeAverage([1, 3, 5]).then(average => console.log("The average is", average));

computeAverage([0.3, 8, 20.5]).then(average => console.log("The average is", average));

But what if we use the empty array?

computeAverage([]).then(average => console.log("The average is", average));

Because the length of the empty array is 0, we are dividing by 0, and an error will be thrown. Notice the error message reads “Uncaught (in promise)”… we can use the catch() method to capture this error, i.e.:

computeAverage([])
    .then(average => console.log("The average is", average))
    .catch(err => console.error("Encountered an error:", err));

Now when we run this code, the error is handled by our catch(). We’re still printing it to the console as an error - but notice the message now reads "Encountered an error" ..., i.e. it’s our error message!

Let’s try one more - an array that cannot be averaged, i.e.:

computeAverage(['A', 'banana', true])
    .then(average => console.log("The average is", average))
    .catch(err => console.error("Encountered an error:", err));

Here we see the promise resolves successfully, but the result is NaN (not a number). This is because that is the normal result of this operation in JavaScript. But what if we want that to be treated as an error? That’s where the reject() callback provided to the promise comes in - it is used to indicate the promise should fail. We’ll need to rewrite our computeAverage() method to use this:

function computeAverage(numbers)
{
    return new Promise((resolve, reject) => {
        // Sum the numbers
        var sum = numbers.reduce((acc, value) => acc + value);
        // Compute the average 
        var average = sum / numbers.length;
        if(isNaN(average)) reject("Average cannot be computed.");
        else resolve(average);
    });
}

Rejected promises are also handled by the catch() method, so if we rerun the last example:

computeAverage(['A', 'bannana', true])
    .then(average => console.log("The average is", average))
    .catch(err => console.error("Encountered an error:", err));

Notice we now see our error message!

Chaining Promise.prototype.then() and Promise.prototype.catch()

Where Promise.prototype.then() and Promise.prototype.catch() really shine is when we chain a series of promises together. Remember our callback hell example?

webapp.get('/login', (req, res) => {
    parseFormData(req, res, (form) => {
        var username = form.username;
        var password = form.password;
        findUserInDatabase(username, (user) => {
            encryptPassword(password, user.salt, (hash) => {
                if(hash === user.passwordHash) 
                    res.setCookie({user: user});
                    res.end(200, "Logged in successfully!");
                else
                    res.end(403, "Unknown username/password combo");
            });
        });
    });
});

If each of our methods returned a promise, we could re-write this as:

webapp.get('/login', (req, res))
    .then(parseFormData)
    .then(formData => {
        var username = formData.username;
        var password = formData.password;
        return username;
    })
    .then(findUserInDatabase)
    .then(user => {
        return encryptPassword(password, user.salt);
    })
    .then(hash => {
        if(hash === user.passwordHash)
            res.setCookie({user: user});
            res.end(200, "Logged in successfully");
        else 
            res.end(403, "Unknown username/password combo");
    })
    .catch(err => {
        res.end(500, "A server error occurred");
    });

Promise.All()

In addition to processing promises in serial (one after another) by chaining .then() methods, sometimes we want to do them in parallel (all at the same time). For example, say we have several independent processes (perhaps each running on a webworker or separate Node thread) that when finished, we want to average together.

The Promise.All() method is the answer; it returns a promise to execute an arbitrary number of promises, and when all have finished, it itself resolves to an array of their results.

Let’s do a quick example using this method. We’ll start by declaring a function to wrap a fake asynchronous process - basically creating a random number (between 1 and 100) after a random number of seconds (between 0 and 3):

function mockTask() {
    return new Promise((resolve, reject) => {
        setTimeout(() => {
            var value = Math.ceil(Math.random()*100);
            console.log("Computed value", value);
            resolve(value);
        }, Math.random() * 3000)
    });
}

Now let’s say we want to compute an average of the results once they’ve finished. As Promise.All() returns a Promise that resolves to a an array of the results, we can invoke our computeAverage() (which we declared previously) in a chained .then():

Promise.all([
    mockTask(),
    mockTask(),
    mockTask(),
    mockTask()
])
.then(computeAverage)
.then(console.log);

Note that because computeAverage takes as a parameter an array, and console.log takes as its parameter a value, and those are what the previous promises resolve to, we don’t have to define anonymous functions to pass into .then() - we can pass the function name instead.

Many JavaScript programmers found this format more comfortable to write and read than a series of nested callbacks. However, the async and await syntax offers a third option, which we’ll look at next.

Async and Await

The async and await keywords are probably more familiar to you from languages like C#. JavaScript introduced them to play much the same role - a function declared async is asynchronous, and returns a Promise object.

With this in mind, we can redeclare our createTimer() method using the async keyword:

async function createTimer(milliseconds) {
    return new Promise((resolve, reject) => {
        setTimeout(() => {
            resolve();
        }, milliseconds);
    });
}

Now, instead of using the promise directly, we can use the await keyword in other code to wait on the promise to resolve, i.e.:

await createTimer(4000);
console.log("Moving on...");

Try running this in the console. Notice that the second line is not executed until the timer has elapsed after 4 seconds!

Similarly, if we need to use a value computed as part of an asynchronous process, we can place the await within the assignment. I.e. to reproduce the Promise.All() example in the previous section, we could re-write our mockTask() and computeAverage() as async functions:

async function mockTask() {
    return new Promise((resolve, reject) => {
        setTimeout(() => {
            var value = Math.ceil(Math.random()*100);
            console.log("Computed value", value);
            resolve(value);
        }, Math.random() * 3000)
    });
}

async function computeAverage(numbers)
{
    return new Promise((resolve, reject) => {
        // Sum the numbers
        var sum = numbers.reduce((acc, value) => acc + value);
        // Compute the average
        var average = sum / numbers.length;
        if(isNaN(average)) reject("Average cannot be computed.");
        else resolve(average);
    });
}

And then the code to perform the averaging could be written:

var numbers = [];
numbers.push(await mockTask());
numbers.push(await mockTask());
numbers.push(await mockTask());
numbers.push(await mockTask());
var average = await computeAverage(numbers);
console.log(average);

Many imperative programmers prefer the async and await syntax, because execution of the code pauses at each await, so code statements are executed in the order they appear in the code. However, the actual execution model it is still the event-based concurrency that we introduced with callbacks. Thus, when awaiting a result, the JavaScript interpreter is free to process other incoming events pulled off the event loop.

Summary

In this chapter we learned about many of the approaches and challenges involved in concurrent programming, including asynchronous programming. JavaScript adopts the asynchronous approach through its use of the event loop and queue, allowing asynchronous processes to be invoked, processed on separate threads, and posting their results as new messages on the event queue to be processed when the main thread gets to them.

We saw how this approach allows for multi-threaded programs in the browser through the use of web workers, each of which runs a separate JavaScript interpreter with its own event loop and queue. We also saw how communication between web workers and the main thread are handled through message passing, and how very large data buffers can be transferred instead of copied between these threads for efficiency.

Finally, we examined the callback mechanism used for asynchronous processing in JavaScript, and explored two common abstractions used to make it more programmer-friendly: promises and the async/await keywords.

In the next chapter, we’ll explore Node.js, a server-side implementation of the JavaScript engine that makes heavy use of the JavaScript asynchronous model.

Introduction

Node is an open-source, cross-platform JavaScript runtime environment build on Google’s V8 engine. It was created by Ryan Dahl in 2009 to allow for server-side scripting in JavaScript.

ECMAScript Support

Node supports most of the features of ECMAScript 2015 (ES6), with the notable exception of ES6 modules (as Node adopted the CommonJS module approach before the ES6 proposal, and the two approaches are not interchangeable). You can learn more about Node’s ES6 support here.

Differences between Browser JavaScript and Node

While JavaScript in the browser has been deliberately sandboxed and is only allowed to interact with the current document and the Internet, Node can interact with the file system and operating system of the hosting computer. Also, because Node is not hosted in a browser, it has no Document Object Model (DOM), and no document or window global variables. To reflect this, the global namespace for Node is global instead of window.

Event Loop & Console

Node adopts an asynchronous event-driven approach to computing, much like JavaScript does in the browser. For example, when we set up a HTTP server in Node, we define a function to call when a HTTP request (an event) is received. As requests come in, they are added to a queue which is processed in a FIFO (first-in, first-out) manner.

In addition to events, Node implements many asynchronous functions for potentially blocking operations. For example, consider file I/O. If you write a program that needs to read a file, but when it attempts to do so the file is already open in another program, your program must wait for it to become available. In the meantime, your program is blocked, and its execution pauses. A real-world analogue would be a checkout line at a store. If the cashier is ringing up a customer and finds an item without a price tag, they may leave their station to go find the item on the shelves. While they are away, the line is blocked, everyone in line must wait for the cashier to return.

There are two basic strategies to deal with potentially blocking operations - multi-threading and asynchronous processing. A multi-threading strategy involves parallelizing the task; in our store example, we’d have additional cash registers, cashiers, and lines. Even if one line is blocked, the others continue to ring up customers. In contrast, asynchronous processing moves the potentially blocking task into another process, and continues with the task at hand. When the asynchronous process finishes, it queues its results for the main process to resume. In our store example, this would be the cashier sending another employee to find the price, suspending the sale on the register, and continuing to check out other customers while the first customer waits. When the other employee returns with a price, the cashier finishes checking out the current customer, then resumes the first customer’s transactions.

Node uses this asynchronous model to handle most potentially blocking operations (and often provides a synchronous approach as well). When possible, the asynchronous process is handled by the operating system, but the Node runtime also maintains a pool of threads to farm tasks out to.

The Node event loop is divided into a series of phases - each queues the associated kinds of events and processes them in a round-robin fashion. This helps ensure that one kind of event can’t overwhelm the system:

For a more detailed breakdown of the Node event loop, check out this blog post by Daniel Khan or the Node Event Loop Documentation.

Console

To produce output on the terminal, Node provides a global instance of a Console called console that is accessible throughout the code. The typical way to produce text output on the terminal in Node is using the console.log() method.

Node actually defines multiple log functions corresponding to different log levels, indicating the importance of the message. These are (in order of severity, least to most):

1. console.debug()
2. console.info()
3. console.warn()
4. console.error()

These are all aliases for console.log() (console.debug() and console.info()) or console.error() (console.warn() and console.error()). They don’t really do anything different, which might lead you to wonder why they exist…

But remember, JavaScript is a dynamic language, so we can re-define the console object with our own custom implementation that does do something unique with these various versions. But because they exist, they can also be used with the built-in console. This way our code can be compatible with both approaches!

Asynchronous Functions

The benefit of the asynchronous approach is that all user-written code runs in a single-threaded environment while avoiding blocking. This means for the most part, we can write code the way we are used to, with a few tweaks for asynchronous functions.

Consider the two approaches for reading and printing the contents of a file, below:

const fs = require('fs');

// Synchronous approach
var data = fs.readFileSync('file.txt');
console.log(data);


// Asynchronous approach
fs.readFile('file.txt', function(err, data){
  console.log(data);
});

In the synchronous function, the contents of the file are returned, and assigned to the variable data. Conversely, in the asynchronous approach the file contents are passed into a callback function that is invoked when the asynchronous process finishes (and when the callback phase of the Node event loop is reached). The function itself returns nothing (undefined in Node). The important difference between the two is in the first approach, the program waits for the file to be read. In the second, the program keeps executing lines of code after the asynchronous call, even though the file hasn’t been read yet.

Asynchronous Callback Structure

In most Node libraries, the callback function provided to the asynchronous function follows the same format - the first parameter is an error value, which will be populated with text or object if an error occurred, and otherwise will be a falsy value (a value that evaluates to false, like false, null, undefined, or 0). Successive arguments are the data the function was intended to retrieve - i.e. the contents of the file in the above example. Of course, if an error was encountered, the later values may be wrong. Thus, most programmers follow the pattern of checking for an error at the start of the callback, and halting execution if one is encountered. Rewriting the example above, we would see:

fs.readFile("file.txt", function(err, data){
  if(err) return console.error(err); 
  console.log(data);
});

If err is a truthy value (any non-falsy value, in this case an Exception object or a string), we log the error to the console and return, halting execution of the rest of the function.

Common Asynchronous Misconceptions

It is very important to understand that the callback is executed at a future point in time, and execution continues to further lines of code. Consider this example:

var contents;
fs.readFile("example.txt", function(err, data) {
  contents = data;
});
console.log(contents);

Assuming the file example.txt contains only the line "hello world", what do you think is printed?

You might think that it would be "hello world", but the console.log(data) happens before the callback function is executed, so it will be undefined. If you wanted to print the file contents, you would have to instead do something like:

var contents;
fs.readFile("example.txt", function(err, data) {
  contents = data;
  console.log(contents);
});

Because the logging now happens inside the callback value, it will only occur after the file has been read, and the results added to the event queue, which is where the contents variable is initialized.

Modules

One major feature Node introduced to JavaScript was the ability to encapsulate code into separate files using modules. The approach adopted by Node is the CommonJS module pattern.

$ node --input-type=module <file>

Writing a Module

A module is simply a JavaScript file (typically saved with a js extension). Code within the module is locally scoped to the module; code intended to be accessed outside of the module must be explicitly exported by assigning it to the module.exports parameter. This example exports a function to print “Hello World”:

/* hello.js */

function helloWorld() {
  console.log("Hello World");
}

module.exports = helloWorld;

You can export a value, function, or object from a module. If you don’t specify an export, a module exports undefined.

Loading a Module

CommonJS Modules are loaded using the require() function, which loads the JavaScript in the module and returns any exports from the module. Objects, functions, or even values can be exported, i.e. this example loads our earlier module, assigns the result to the variable greet, and invokes it:

var greet = require('./hello');
greet();

This example will print “Hello World” to the console.

There are three kinds of modules you can load in Node: 1) core libraries, 2) npm packages, and 3) user-authored modules.

Core libraries are part of Node. Examples are the fs (file system) library, the http library, and the crypto library. A full list of libraries can be found in the Node documentation. Node libraries are required using their name as as the argument, i.e. require('fs') will require the filesystem library.

Npm packages are typically open-source libraries created by the community and are downloaded using the Node Package Manager (npm). These are installed either globally, or within a node_modules folder created by npm within your project. Npm modules are required by their package name, i.e. require('webpack') will load the webpack package.

User-written modules are loaded from the filesystem, and are typically located in your project directory or subdirectories within it. They are required using the filepath to their file, i.e. require('./hello.js). It is important to use the period-slash (./) indicating current directory to distinguish files from npm packages.

Packages

The Node Package Manager allows you to create a package representing your project. This is similar to Visual Studio’s idea of a project - a package is a complete Node program.

Just as Visual Studio adds solution and project files, a Node package adds a file named package.json and a directory named node_modules.

The Package File

Every node package has in its top-level directory a file named package.json. This JSON file provides important information about the project, including:

• The name of the project
• The version of the project
• The author of the project
• The entry point (the file that should be processed first, much like a C# Program.cs file or C++ main.cpp file). Owing to its roots as a language for developing webservers, the default is index.js, though it can be anything you want (many projects use server.js or main.js).
• Any scripts associated with the project (often we’ll have scripts to run and test the project)
• Any dependencies of the project (these are additional packages that are copied into the node_modules file)
• The home repository of the project (often a GitHub repository, though other options are possible)
• The license under which this package is released

Most of the properties of the package.json are optional, and there are options I have listed above are not comprehensive. The package.json object format is described in great detail in the npm documentation.

Semantic Versioning

Node packages use semantic versioning, a numbering scheme that uses three numbers separated by periods in the pattern MAJOR.MINOR.PATCH. For example, your Codio Box is likely running Ubuntu 18.04.3 - that is Ubuntu, major release 18, minor release 4, patch 3. If its developers found a bug or security vulnerability and fixed it, they would release a new patch version, i.e. Ubuntu 18.04.4. If they made some improvements that don’t change the way you work with the operating system, those might be a minor release, i.e. Ubuntu 18.05.0 (note that the patch release number starts over at 0). And if a major change in the way the software works was made, that would be a major release, i.e. Ubuntu 19.0.0. Additionally, you probably have seen the program listed as Ubuntu 18.04.4 LTS. The LTS is not part of semantic versioning, but indicates this is a long term support release, i.e. one the developers have committed to fixing problems with for a specific duration of time.

When releasing your own Node packages, you should strive to practice using semantic versioning. That way you get into the habit, and it will be easier when you become a professional developer. Another good practice is to have a changelog - a text or markdown file that lists the changes made in each successive version of the program, usually as a bulleted list.

Initializing the Package

You can manually create the package.json file, but npm also provides a wizard to make this process easier. From the terminal, while at the root directory of your project, enter the command:

$ npm init

This will launch the wizard, asking you a series of questions used to create the project. It will also offer default options for many of the values; you can simply press the space bar to accept these. Moreover, if your directory contains an already-initialized git repository, it will use that repository’s origin as the basis for the repository option. The wizard will create the package.json file in the project’s root directory.

Next we’ll look at some aspects of the package in more detail.

Dependencies

A second great benefit of creating your project as a Node package is that dependencies can be managed using the Node Package Manager (npm). You can install any Node package with the command $npm install [package name]. This command looks for the corresponding package in an online repository, and if it is found, downloads it and saves it to the subdirectory node_modules in your package directory.

It also creates an entry in the package.json file corresponding to the package you installed, and also an entry in the package.lock.json file. The entry in your package.json file may specify a specific version, i.e.:

{
    "name": "example-package",
    "version": "1.0.0",
    "dependencies": {
        "foo": "2.1.3",
        "bar": "^1.1.0",
        "doh": "~4.3.2"
    }
}

The ^ before the "bar" dependency indicates that npm can use any minor release after this version, i.e. we could use bar 1.2.1 but not bar 2.1.1. The ~ before the "doh" dependency indicates that npm can use any patch release after this version, i.e. doh 4.3.4 but not doh 4.4.0. Because we specified the exact version for "foo", it will always install foo 2.1.3. We could also indicate we will accept any major version with an *, but this is rarely used. Additionally, we can specify a git repo or a location on our hard drive, though these approaches are also not commonly used.

The reason we want the dependency information specified this way is that when we commit our package to source control, we typically exclude the dependencies found in node_modules. As this folder can get quite large, this saves us significant space in our repository. When you clone your package to a new location, you can re-install the dependencies with the command:

$ npm install 

This will pull the latest version of each dependency package allowable. Additionally, some modules may have their own dependencies, in which case npm will strive to find a version that works for all requirements.

Finally, the package.lock.json contains the exact version of the dependencies installed. It is intended to be committed to your repository, and if it exists it will make the npm install command install the exact same packages. This can help avoid problems where two versions of a package are slightly different.

Development Dependency

In addition to regular dependencies, we can specify dependencies we only need during development. Adding the --save-dev flag to the command will add the package to a development dependency list in your package.json file. Development dependencies are only installed in the development environment; they are left out in a production environment.

Git

Most Node packages are made available as git repositories, and npm has built-in support for using git.

The Repository Property

In your package.json file, you can specify a "repository" property, which specifies where the repository for this project exists. Consider the following example of the npm command-line interface package:

"repository": {
  "type" : "git",
  "url" : "https://github.com/npm/cli.git"
}

For many open-source projects, the repository is located on Github, a GitHub gist, BitBucket, or a GitLab instance. These can be specified with a shorthand, which matches the corresponding npm install argument:

"repository": "npm/npm"

"repository": "github:user/repo"

"repository": "gist:11081aaa281"

"repository": "bitbucket:user/repo"

"repository": "gitlab:user/repo"

If your project folder already is set up as a git repo before you run $npm init, then the wizard will suggest using the existing repository.

The .gitignore File

Like any git repo, there are some files you will want to exclude from version control. The most important of these is the node_modules folder, which we exclude for two reasons:

1. The folder’s contents are large, and are available from the original sources specified in your package’s dependencies - therefore they represent significant and unnecessary glut in a repository.
2. Some portions of your dependencies may be compiled from C code as they are installed. Since C code is compiled into binaries for a specific target machine’s architecture, you will encounter errors if your development and production machines are different. Re-installing packages in the production machine with $npm install avoids this.

Therefore, you should include at least this line in your .gitignore file:

node_modules/

It’s also a good idea to leave out any logging files:

logs
*.log

You may also want to look at GitHub’s boilerplate Node .gitignore template, which adds additional rules based on many popular Node frameworks (adding an ignore for a file or directory that doesn’t exist in your project is harmless).

The .gitignore file should ideally be added before the repository has been created, as files already under version control override .gitignore rules.

Initializing the Repository

Now we can initialize our repository with the $git init command in the terminal:

$ git init 

Note how a .git directory is added to our project? This is where git stores the diff information that allows it to track versions of our code.

The init command adds the current directory structure (less that specified in the .gitignore file) to the staging list. Otherwise, we’d need to stage these files ourselves for version tracking with the git command, add .:

$git add .

Once staged, we can commit the initial version of our package with a git commit command:

$ git commit -a -m "Initial commit"

The -a flag indicates we are committing all currently staged files, alternatively you can commit files one-at-a-time, if you only want to commit one or a few files. The -m file indicates we are also adding a commit message, which follows in quotes (""). If we don’t use this flag, then git will open a text editor to write the commit message in. For the Linux platform, the default is vi, a powerful command line text editor that has a bit of a learning curve. If you find vi has launched on you and you are uncertain how to proceed, follow these steps:

1. Type your commit message
2. Hit the [ESC] key - this moves you from edit mode to command mode
3. Type the command :wq. Vi commands always start with a colon (:), and can be grouped. The w indicates the file should be written (saved), and the q indicates we want to quit the editor.

Adding Remote Repositories

If you want to host your repository on GitHub or another hosting tool (such as BitBucket or GitLab), you will need to add that repository’s url as a remote origin for your local repository. You can do so with the command:

$ git remote add origin [repo url]

Where [repo url] is the url of the remote repo. If you want to use GitHub, create an empty repository, and copy the created repository’s link.

If you clone a repository from GitHub or another tool, the remote origin is automatically set to that parent repository. If you ever need to change the remote origin, it’s a two-step process:

1. Delete the current origin with the git command:$git remote rm origin
2. Add your new origin with the git command: $git remote add origin

You can also add other targets than origin (origin is a generic target meaning the originating repository). To see all remote repositories, use the command $git remote -v, which lists all registered remote repositories (the -v flag is for verbose).

Pushing and Pulling

The $git push and $git pull commands are shorthand for pushing and pulling the master branch to and from the origin repository. The full commands can be used to push an alternate branch or to reach an alternate remote repository: $git push [remote repo name] [branch name], $git pull [remote repo name] [branch name]

Git and Codio

Pushing your code to a remote repository is a good way to move it between Codio boxes. You can clone such a repository after you create it with the command:

$ git clone [git clone url]

Where the [git clone url] is the URL of your remote repository. Cloning a repository automatically sets its remote origin repo to the repository it was cloned from.

Summary

Node, and its package manager, npm, are powerful tools for developing server-side applications in JavaScript. In the past chapter, we’ve discussed the event loop that drives Node’s asynchronous, event-driven approach, and the asynchronous function pattern employed by many node libraries. We’ve also talked about how Node.js code can be organized into modules, how those modules can be imported with a require() call, and how the node package manager (npm) can be used to download and install open-source packages locally. Finally, we’ve discussed how to create our own local packages from our projects.

Resources

Node can be downloaded from nodejs.org or installed using Node Version Manager (nvm).

The latest Node documentation can be found at https://nodejs.org/en/docs/

Introduction

The first web servers were developed to fulfill a simple role - they responded to requests for HTML documents that were (hopefully) located in their physical storage by streaming the contents of those documents to the client.

This is embodied in our request-response pattern. The client requests a resource (such as a HTML document), and receives either a status 200 response (containing the document), or an error status code explaining why it couldn’t be retrieved.

HTTP in Node

Node was written primarily to provide tools to develop web servers. So it should come as no surprise that it supports HTTP through a built-in library, the http module. This module provides support for creating both web servers and web clients, as well as working with http requests and responses. Let’s start by examining the latter.

Node HTTP Request

Remember that a HTTP request is nothing more than a stream of text formatted according to the HTTP standards, as we discussed in Chapter 2. Node makes this information easier to work with in your server code by parsing it into an object, an instance of http.IncomingMessage. Its properties expose the data of the request, and include:

• message.headers - the request headers as a JavaScript object, with the keys corresponding to the HTTP header key, and the values corresponding to the HTTP header values.
• message.method - request method, i.e. 'GET', 'POST', 'PUT', 'PATCH', or 'DELETE'
• message.url - the URL string provided through the request. Note that this URL does not contain the protocol and host information (those were set during the connection opening process)

You typically won’t create an http.IncomingMessage, rather it will be provided to you by an instance of http.Server, which we’ll talk about shortly.

Node HTTP Response

The HTTP response is also nothing more than a stream of text formatted according to the HTTP standards, as laid out in Chapter 2. Node also wraps the response with an object, an instance of http.ServerResponse. However, this object is used to build the response (which is the primary job of a web server). This process proceeds in several steps:

1. Setting the status code & message
2. Setting the headers
3. Setting the body
4. Setting the trailers
5. Sending the response

Not all of these steps are required for all responses, with the exception of the sending of the response. We’ll discuss the process of generating a response in more detail later. As with the http.IncomingMessage, you won’t directly create http.ServerResponse objects, instead they will be created by an instance of http.Server, which we’ll look at next.

Node HTTP Server

The http.Server class embodies the basic operating core of a webserver, which you can build upon to create your own web server implementations. Instances of http.Server are created with the factory method http.createServer(). This method takes two parameters, the first is an optional JavaScript object with options for the server, and the second (or first, if the options parameter is omitted) is a callback function to invoke every time the server receives a request. As you might expect, the method returns the created http.Server instance.

The callback function always takes two parameters. The first is an instance of http.IncomingMessage (often assigned to a variable named req) that represents the request the server has received, and the second is an instance of http.ServerResponse (often assigned to available named res) that represents the response the server will send back to the client, once you have set all its properties. This callback is often called the request handler, for it decides what to do with incoming requests. Each incoming request is handled asynchronously, much like any event.

Once the server has been created, it needs to be told to listen for incoming requests. It will do so on a specific port. This port is represented by a number between 0 and 65535. You can think of your computer as an apartment complex, and each port number corresponding to an apartment number where a specific program “lives”. Thus, to send a message to that particular program, you would address your letter to the apartment address using the apartment number. In this analogy, the apartment address is the host address, i.e. the IP address of your computer, and the specific apartment is the port. Certain port numbers are commonly used by specific applications or families of applications, i.e. web servers typically use port 80 for HTTP and port 443 for HTTPS connections. However, when you are developing a webserver, you’ll typically use a non-standard port, often an address 3000 and up.

Thus, to start your webserver running on port 3000, you would invoke [server.Listen()] with that port as the first argument. An optional second argument is a callback function invoked once the server is up and running - it is usually used to log a message saying the server is running.

An Example Server

A quick example of a server that only responds with the message “Hello web!” would be:

const http = require('http');
const server = http.createServer((req, res) => {
    req.end("Hello web!");
});
server.listen(3000, () => {
    console.log("Server listening at port 3000");
});

Once the server is running, you can visit http://localhost:3000 on the same computer to see the phrase “Hello web!” printed in your browser.

The localhost hostname is a special loopback host, which indicates instead of looking for a computer on the web, you’re wanting to access the computer you’re currently on. The :3000 specifies the browser should make the request to the non-standard port of 3000. If we instead used port 80, we could omit the port in the URL, as by default the browser will use port 80 for http requests.

Request Handling

An important aspect to recognize about how Node’s http library operates is that all requests to the server are passed to the request handler function. Thus, you need to determine what to do with the incoming request as part of that function.

Working with the Request Object

You will most likely use the information contained within the http.IncomingMessage object supplied as the first parameter to your request handler. We often use the name req for this parameter, short for request, as it represents the incoming HTTP request.

Some of its properties we might use:

• The req.method parameter indicates what HTTP method the request is using. For example, if the method is "GET", we would expect that the client is requesting a resource like an HTML page or image file. If it were a "POST" request, we would think they are submitting something.

• The req.url parameter indicates the specific resource path the client is requesting, i.e. "/about.html" suggests they are looking for the “about” page. The url can have more parts than just the path. It also can contain a query string and a hash string. We’ll talk more about these soon.

• The req.headers parameter contains all the headers that were set on this particular request. Of specific interest are authentication headers (which help say who is behind the request and determine if the server should allow the request), and cookie headers. We’ll talk more about this a bit further into the course, when we introduce the associated concepts.

Generally, we use properties in combination to determine what the correct response is. As a programmer, you probably already realize that this decision-making process must involve some sort of control flow structure, like an if statement or switch case statement.

Let’s say we only want to handle "GET" requests for the files index.html, site.css, and site.js. We could write our request handler using both an if else statement and a switch statement:

function handleRequest(req, res) {
    if(req.method === "GET") {
        switch(req.url) {
            case "/index.html":
                // TODO: Serve the index page 
                break;
            case "/site.css":
                // TODO: Serve the css file 
                break;
            case "/site.js":
                // TODO: Serve the js file
                break;
            default:
                // TODO: Serve a 404 Not Found response
        }
    } else {
        // TODO: Serve a 501 Not Implemented response
    }
}

Notice that at each branching point of our control flow, we serve some kind of response to the requesting web client. Every request should be sent a response - even unsuccessful ones. If we do not, then the browser will timeout, and report a timeout error to the user.

Working with the Response Object

The second half of responding to requests is putting together the response. You will use the http.ServerResponse object to assemble and send the response. This response consists of a status code and message, response headers, and a body which could be text, binary data, or nothing.

There are a number of properties and methods defined in the http.ServerResponse to help with this, including:

• ServerResponse.statusCode can be used to manually set the status code
• ServerResponse.statusMessage can be used to manually set the status message. If the code is set but not the message, the default message for the code is used.
• ServerResponse.setHeader() adds a header with the supplied name and value (the first and second parameters)
• ServerResponse.end() sends the request using the currently set status and headers. Takes an optional parameter which is the response body; if this parameter is supplied, an optional second parameter specifying the encoding can also be used. A final optional callback can be supplied that is called when sending the stream is complete.

Consider the 501 Not Implemented response in our example above. We need to send the 501 status code, but there is no need for a body or additional headers. We could use the req.statusCode property to set the property, and the req.end() method to send it:

    // TODO: Serve a 501 Not Implemented response
    res.status = 501;
    res.end();

The sending of a response with a body is a bit more involved. For example, to send the index.html file, we would need to retrieve it from the hard drive and send it as the body of a request. But as the default status code is 200, we don’t need to specify it. However, it is a good idea to specify the Content-Type header with the appropriate mime-type, and then we can use the res.end() method with the file body once we’ve loaded it, i.e.:

    // TODO: Serve the index page 
    fs.readFile('index.html', (err, body) => {
        if(err) {
            res.status = 500;
            res.end();
            return;
        }
        res.setHeader("Content-Type", "text/html");
        res.setHeader("Content-Length", body.length);
        res.end(body, "utf8");
    });

Notice too, that we need to account for the possibility of an error while loading the file index.html. If this happens, we send a 500 Server Error status code indicating that something went wrong, and it happened on our end, not because of a problem in the way the client formatted the request. Notice too that we use a return to prevent executing the rest of our code.

We also supply the length of the response body, which will be the same as the buffer length or the length of a string sent as the body. Binary data for the web is counted in octets (eight bits) which conveniently is also how Node buffers are sized and the size of a JavaScript character.

Chaining writeHead() and end()

The http.ServerResponse object also has a method writeHead() which combines the writing of status code, message, and headers into a single step, and returns the modified object so its end() method can be chained. In this way, you can write the entire sending of a response on a single line. The parameters to response.writeHead() are the status code, an optional status message, and an optional JavaScript object representing the headers, using the keys as the header names and values as values.

Serving the css file using this approach would look like:

    // TODO: Serve the site css file 
    fs.readFile('site.css', (err, body) => {
        if(err) return res.writeHead(500).end();
        res.writeHead(200, {
            "Content-Type": "text/html",
            "Content-Length": body.length
        }).end(body, "utf8");
    });

You can use any combination of these approaches to send responses.

The Importance of Being Async

You may have noticed that we used the asynchronous version of fs.readFile() in our response handler. This is critical to good performance with a Node-based webserver - any potentially blocking action taken in the request handler should be asynchronous, because all incoming requests must be processed on the same thread. If our event loop gets bogged down handling a blocked process, then nobody gets a response!

Consider if we implemented one of the file serving options using the synchronous fs.readFileSync():

    // TODO: Serve the site js file 
    try {
      var body = fs.readFileSync('site.js');
      res.writeHead(200, {
        "Content-Type": "application/javascript",
        "Content-Length": body.length
      }).end(body, "utf8");
    } catch(err) {
        res.writeHead(500).end();
    }

It does not look that different from the asynchronous version, but consider what happens if site.js cannot be opened immediately (perhaps it is locked by a system process that is modifying it). With the synchronous version, we wait for the file to become available. While we wait, incoming requests are added to our event queue… and none are processed, as the event loop is paused while we are waiting for fs.readFileSync('site.js') to resolve. If it takes more than three seconds, the clients that are waiting will start seeing timeouts in their browsers, and will probably assume our site is down.

In contrast, if we used the asynchronous version, the reading of the file is handed off to another thread when fs.readFile() is invoked. Any additional processing to do within the event loop for this request is finished, and the next task is pulled from the event queue. Even if our request for the original file never completes, we still are serving requests as quickly as we get them.

This asynchronous approach is key to making a Node-based website perform and scale well. But it is vitally important that you, as a web developer, understand both why using asynchronous processes is important, and how to use them correctly. Because if you don’t, your application performance can be much, much worse than with other approaches.

Caching for the Win

In our example web server, we argued that asynchronous file reading was better than synchronous because reading from a file is a potentially blocking operation that can take a long time to perform. But even when it doesn’t block, it can still take a lot of time, making it the most expensive part of our request handling operation in terms of the time it takes to perform.

If we really want to squeeze all the performance we can out of our server (and therefore handle as many users as possible), we need to consider the strategy of caching. This means storing our file content in a location where access is faster - say in memory rather than on the disk. Variables in our application are stored in RAM, even variables whose contents were initialized from disk files. Thus, assigning the contents of a file to a variable effectively caches it for faster access.

Moreover, because Node variables are only accessible from the Node process, which from our perspective in the event loop is single-threaded, we don’t have to worry about blocking once the file has been loaded. One strategy we can employ is to pre-load our files using synchronous operations, i.e.:

const http = require('http');
const fs = require('fs');

const html = fs.readFileSync('index.html');
const css = fs.readFileSync('site.css');
const js = fs.readFileSync('site.js');

Now we can define our revised event handler, which uses the cached versions:

function handleRequest(req, res) {
    if(req.method === "GET") {
        switch(req.url) {
            case "/index.html":
                // Serve the index page 
                res.writeHead(200, {'Content-Type': 'text/html', 'Content-Length': html.length}).end(html);
                break;
            case "/site.css":
                // Serve the css file 
                res.writeHead(200, {'Content-Type': 'text/html', 'Content-Length': css.length}).end(css);
                break;
            case "/site.js":
                // Serve the js file
                res.writeHead(200, {'Content-Type': 'text/html', 'Content-Length': js.length}).end(js);
                break;
            default:
                // Serve a 404 Not Found response
                res.writeHead(404).end();
        }
    } else {
        // Serve a 501 Not Implemented response
        res.writeHead(501).end();
    }
}

Finally, we create and start the server:

var server = http.createServer(handleRequest);
server.listen(80, ()=>{
    console.log("Server listening on port 80");
});

Notice in this server implementation we use the synchronous fs.readFileSync(), and we don’t wrap it in a try ... catch. That means if there is a problem loading one of the files, our Node process will crash. But as these files are loaded first, before the server starts, we should see the error, and realize there is a problem with our site files that needs fixed. This is one instance where it does make sense to use synchronous file calls.

While caching works well in this instance, like everything in computer science it doesn’t work well in all instances. Consider if we had a server with thousands of files - maybe images that were each 500 Megabytes. With only a thousand images, we’d have 500 Gigabytes to cache… which would mean our server would need a lot of expensive RAM. In a case like that, asynchronous file access when the file is requested makes far more sense.

Also, depending on the use pattern it may make sense to cache some of the most frequently requested images. With careful design, this caching can be dynamic, changing which images are cached based on how frequently they are requested while the server is running.

Index Pages

The original purpose of the World-Wide-Web was to share webpages and other digital resources across the Internet. In many ways, an early web server was like a hard drive that was open to the world. Think about the HTTP methods, "GET" is like a file read, "POST" is like a write, "PUT" and "PATCH" like a file modification, and "DELETE" was a file erasure.

So, just like when you browse your hard drive using Windows Explorer or other software, it was necessary for these early web pages to display an index - a listing of all the contents of a directory. You’ve seen similar in Codio if you ever used the “Project Index” option in the run menu - it displays an index of the project directory:

This is a pretty standard auto-generated directory listing - it provides the path to the directory being displayed, and all the contents of the directory as hyperlinks. Clicking on a file will open or download it, and clicking a directory will open that directory’s auto-generated directory listing.

As the use of the web expanded into commercial and public life, many web developers wanted to replace auto-generated directory listing pages with a carefully designed home page. But auto-generated directory listing pages remained an important feature for many sites that served as a more traditional file server.

The compromise adopted by most web servers was that if the directory contained an HTML file named index.html (or sometimes index with any extension, i.e. index.php), that page would be served in leu of an auto-generated index page. Most also allow disabling the directory listing as a configuration option.

Partial Downloads

While we normally think of downloading an entire file from the web, there are some situations where it makes sense to download only part of a file. One case is with a large file download that gets interrupted - it makes a lot of sense to start downloading the remaining bytes from where you left off, rather than starting over again. A second case is when you are streaming media; often the user may not watch or listen to the entire media file, so why download the bytes they don’t need? Or if it is a live stream, we explicitly can’t download the entire thing, because later parts do not yet exist!

Range Headers

HTTP explicitly supports requesting only a part of a resource with the Range header. This allows us to specify the unit of measure (typically bytes), a starting point, and an optional end. This header is structured:

Range: <unit>=<range-start>-<range-end>

Where <unit> is the unit of measure, the <range-start> is the start of the range to send, measured in the provided unit, and <range-end> is the end of the range to send.

Thus, a real-world Range header might look like:

Range: bytes=200-300

You can also specify only the starting point (useful for resuming downloads):

Range: <unit>=<range-start>-

Finally, you can specify multiple ranges separated by commas:

Range: <unit>=<range1-start>-<range1-end>, <range2-start>-<range2-end>, <range3-start>-

Of course, as with all request headers, this indicates a desire by the web client. It is up to the web server to determine if it will be honored.

Partial Content

Which brings us to the 206 Partial Content response status code. If the server chooses to honor a range specified in the request header, it should respond with this status code, and the body of the response should be just the bytes requested.

In addition, the response Content-Range header should be included with the response, specifying the actual range of bytes returned. This is similar to the Range header, but includes a the total size:

Content-Range: <unit> <range-start>-<range-end>/<size>

An asterisk (*) can be used for an unknown size.

Also, the Content-Type header should also be included, and match the type of file being streamed.

If multiple ranges are included in the response, the Content-Type is "multipart/byteranges" and the response body is formatted similarly to multipart form data.

Introduction

While the first generation of webservers was used to serve static content (i.e. files), it was not long before developers began to realize that a lot more potential existed in the technologies of the web. A key realization here is that the resources served by the web server don’t need to exist to be served.

Consider the directory listing from the previous chapter. It is not based on an existing file, but rather is dynamically created when requested by querying the file system of the server. This brings one clear benefit - if we add files to the directory, the next time we request the listing they will be included.

Thus, we can create resources dynamically in response to a request. This is the core concept of a dynamic web server, which really means any kind of web server that generates at least some of its content dynamically. In this chapter, we will explore this class of web server.

CGI Scripts

Before we dig too deeply into dynamic web servers, we should review our technologies used in the web. On the client side, we have HTML, CSS, and JavaScript. Managing communication between the client and server, we have HTTP. But on the server side of the equation, what standard web technologies do we use?

The answer is none. There is no standard server development language. In fact, web servers can be written in almost every programming language. This gives web application developers a tremendous amount of flexibility. Of course, that also means the choice of server-side technologies can quickly become overwhelming.

One technology that emerged to help manage some of this complexity was the Common Gateway Interface (CGI), a web standard that allowed a traditional static webserver to respond to some requests by running a command-line program and piping the output to the requesting client. The CGI standard defined what variables needed to be collected by the web server and how they would be provided to the script.

The script itself could be written in any language that could communicate using stdin and stdout, though in practice most CGI scripts were written using Perl or Bash script. This strategy was popular with the system admins responsible for deploying webservers like Apache, as these scripting languages and command-line programs were familiar tools. While CGI scripts can be used to build dynamic web pages, by far their most common use was to consume forms filled out by a user, often saving the results to a file or database or sending them via email.

CGI scripts are still used today, but they do have some important limitations. First, for each kind of request that needed handled, a new script would need to be written. The open-ended nature of CGI scripts meant that over time a web application become a patchwork of programs developed by different developers, often using different languages and organizational strategies. And since running a CGI script typically means starting a separate OS process for each request, the CGI approach does not scale well to web applications in high demand.

Thus, web developers began seeking new strategies for building cohesive web servers to provide rich dynamic experiences.

Server Pages

The CGI scripting approach eventually evolved into a concept known as server pages and embodied in the technologies of PHP and Microsoft’s Active Server Pages (ASP), as well as Java Server Pages, and many other less-well-known technologies. While each of these use different scripting languages, the basic idea is the same: Take a traditional static webserver functionality, and couple it with a script interpreter. When most files are requested from the server, the file is served using the same techniques we used in the last chapter. But when a script file understood by our augmented server is requested, the script file is executed, and its output is sent as the response.

This may sound a lot like the CGI Scripts we discussed before, and it certainly is, except for two major innovations. The first of these was that the same interpreter, and therefore OS process, could be used for all requests. As server hardware adopted multiple CPUs and multi-core CPUs, additional interpreter processes were added, allowing incoming requests to be responded to concurrently on separate processors.

Embedded Scripts

The second big innovation innovation was the idea of embedding script directly in HTML code. When the server page is interpreted, the embedded scripts would execute, and their output would be concatenated directly into the HTML that was being served.

For example, a PHP page might look like this:

<!DOCTYPE html>
<html>
  <head><title>PHP Example</title></head>
  <body>
    <h1>A PHP Example</h1>
    <?php
      echo date('D, d M Y H:i:s');
    ?>
  </body>
</html>

Notice everything except the <?php ... ?> is perfectly standard HTML. But when served by an Apache server with mod_php installed, the code within <?php ... ?> would be executed, and its output concatenated into the HTML that would then be served (the echo function prints output, and date() creates the current time in the specified format).

Similarly, an ASP page doing the same task would look like:

<!DOCTYPE html>
<html>
  <head><title>ASP Example</title></head>
  <body>
    <h1>An ASP Example</h1>
    <%
      Response.Write Now()
    %>
  </body>
</html>

These files would typically be saved with a special extension that the server would recognize, i.e. .php for PHP files, and .asp for ASP files. Allowing scripts to be directly embedded within HTML made web development with these tools far faster, and as IDEs were adapted to support syntax highlighting, code completion, and other assistive features with these file types also helped prevent syntax errors in both the HTML and script code.

Active Server Pages and PHP remain commonly used technologies today, and a large portion of legacy websites built using them are still in service. In fact, your personal web space on the CS department server is running in an Apache server set up to interpret PHP files - you could add a PHP script like the one above and and it would execute when you visited the page. You can visit the support Personal Web Pages entry for details on doing so.

Custom Servers

While CGI Scripts and Server Pages offered ways to build dynamic websites using off-the shelf web server technologies (Apache, IIS), many programmers found these approaches limiting. Instead, they sought to build the entire web server as a custom program.

Node was actually developed for exactly this approach - it provides abstractions around the fundamental aspects of HTTP in its http library. This library handles the listening for HTTP requests, and parses the request and uses it to populate a http.ClientRequest object and provides a http.ServerResponse object to prepare and send the response. These are abstractions around the HTTP Request and Response we discussed in chapter 2.

The authors of Node took advantage of the functional nature of JavaScript to make writing servers easy - we just need to provide a function to the server that takes a request/response pair. This has been our handleRequest() method, and we can pack it with custom logic to determine the correct response for the incoming request.

We’ve already written a number of functions that take in a request and response object, and determine the correct response to send. This includes our serveFile(), and our listDirectory(). We can create dynamic pages in much the same way - by writing a custom function to serve the dynamic page.

For example, a very simple dynamic page could be created with this function:

function serveTime(req, res) {
    var html = `
        <!doctype html>
        <html>
          <head>
            <title>Server Time</title>
          </head>
          <body>
            <h1>Time on the server is ${new Date()}</h1>
          </body>
        </html>
    `;
    req.writeHead(200, {
        "Content-Type": "text/html",
        "Content-Length": html.length
    }).end(html);
}

But for a server to be dynamic, we typically want more than just the ability to render a page from code. We also want to interact with the user in some meaningful way - and that means receiving data from them. In the HTTP protocol there are several ways for the user to send data:

1. By the path of the request, i.e. when we request https://support.cs.ksu.edu/CISDocs/wiki/Main_Page the path CISDocs/wiki/Main_Page indicates the page we are requesting

2. By additional information contained in the URL, specifically the query string (the part of the URL that follows the ?) and the hash string (the part of hte URL that follows the #)

3. By information contained in the body of the request

We’ve already looked at the path portion of the URL in describing how we can implement a file server to serve files and directories. But what about the query string and body of the request? Let’s look at how those are handled in Node next.

Query and Hash Strings

Query strings (aka search strings) are the part of the URL that appear after the ? and before the optional #. The hash string is the portion of the url after the #. We’ve mentioned them a bit before, but as we dig into dynamic web servers it makes sense to do a deeper dive, as this is where they really come into play.

The Hash String

First, let’s briefly visit the hash string. It’s traditional use is to indicate a HTML element on the page the browser should scroll to when visiting the URL. When used this way, its value is the id attribute of the element. The browser will automatically scroll to a position that displays the element on-screen. Thus, the traditional use of the hash string is for “bookmarked” links that take you to an exact section of the page, like this one which takes you to a description of your personal CS web space: https://support.cs.ksu.edu/CISDocs/wiki/Personal_Web_Pages#Dynamic_Content.

With the advent of single-page apps, it has come to serve additional purposes; but we’ll talk about those when we visit that topic in a future chapter.

The Query/Search String

Now, the query string is a vital tool for dynamic web pages, as it conveys information beyond the specific resource being asked for to the server. Consider what a dynamic webserver does - it takes an incoming request, and sends a response. In many ways, this is similar to a function call - you invoke a function, and it returns a value. With this in mind, the path of the URL is like the function name, and the query string becomes its parameters.

Much like parameters have a name and a value, the query string is composed of key/value pairs. Between the key and value of each pair is a =, and between the pairs is a &. Because the = and & character now have a special meaning, any that appear in the key or value are swapped with a special percent value, i.e. & becomes %26 and = becomes %3D. Similarly, other characters that have special meanings in URLs are likewise swapped. This is known as URL or percent encoding, and you can see the swapped values in the table below:

In JavaScript, the encodeURIComponent() function can be used to apply percent encoding to a string, and the decodeURIComponent() can be used to reverse it.

Node offers the querystring library. It offers a querystring.parse() method which can parse a query string into a JavaScript object with the decoded key/value pairs transformed into properties and a querystring.stringify() method which can convert a JavaScript object into an encoded query string. In addition, these methods allow you to override what characters are used as delimiters, potentially allowing you to use alternative encoding schemes.

Forms and the Query String

While we can type a URL with a query string manually, by far the most common way to produce a query string is with a <form> element. When a form is submitted (the user clicks on a <input> with type attribute of "submit"), it is serialized and submitted to the server. By default, it will be sent to the same URL it was served from, using a GET request, with the fields of the form serialized as the query string.

This is why one name for the query string is the search string, as it is commonly used to submit search requests. For example, Google’s search page contains a form similar to this (very simplified form):

<form>
    <input name="q" type="text">
    <input type="submit" value="Google Search">
</form>

Notice the form has one text input, named "q". If we type in search terms, say “k-state computer science” and click the search button, you’ll notice in the resulting url the query string contains q=k-state+computer+science (plus a lot of other key/value pairs). If you type "https://www.google.com/search?q=k-state+computer+science" into the address bar, you’ll get the same page.

Notice though, that google uses www.google.com/search as the URL to display search results at, instead of the original www.google.com page. We can have a form submit to a different URL than it was served on by setting its action attribute to that URL.

We can also change the HTTP method used to submit the form with the method attribute. By default it uses a GET request, but we can instead set it to a POST request. When a POST request is used, the form is no longer encoded in the query string - it is instead sent as the body of the POST request.

Form Encoding

When a form is submitted as a POST request, its inputs are serialized according to the form’s encoding strategy. This value is also used as the Content-Type header of the request. The three form encoding values are:

• application/x-www-form-urlencoded (the default)
• multipart/form-data (used for file uploads)
• text/plain (used for debugging)

The <form> element’s enctype attribute can be set to any of these three possible values.

application/x-www-form-urlencoded

This serialization format consists of key/value pairs where the keys and values are url (percent) encoded, and between each key and value is a = and between each pair is a &. In other words, it is the same encoding that is used in the query string - the only difference is the query string begins with a ?.

Thus, the form:

<form method="post">
    <input type="text" name="first" placeholder="Your first name">
    <input type="text" name="middle" placeholder="Your middle initials">
    <input type="text" name="last" placeholder="Your last name">
    <input type="number" name="age" placeholder="Your age">
    <input type="submit" value="Save My Info">
</form>

When submitted by John Jacob Jingleheimer Schmidt would be encoded:

first=John&middle=J.+J.&last=Schmidt&age=30

If we parsed this string with querystring.parse() the resulting JavaScript object would be:

{
    first: "John",
    middle: "J. J.",
    last: "Schmidt",
    age: "30"
}

multipart/form-data

There is one limitation to the application/x-www-form-urlencoded encoding strategy - there is no way to include binary data, i.e. files. If you submit a form that includes an <input> of type file and encode it with application/x-www-form-urlencoded, the value supplied by the <input type="file"> will be the filename only. The actual body of the file will not be available. The reason is simple - there’s no easy way to put raw binary data into a text file.

The multipart/form-data solves this in a less-than easy way. It splits the body into blocks - one for each input. Between each block is a sequence of bytes used as a separator. These boundary bytes cannot be found in any of the file data (as if it were, that block would be split up). And each block has its own header section and body section. The header defines what the input is, and what its body contains. If it was a non-file input, it will be text, and if it was a file input, the content will be the raw bytes, encoded in base64.

When the server receives this body, it must use the boundary bytes to separate the blocks, and then parse each block. There are no built-in libraries in Node for doing this, though there are many npm packages that do. We’ll also write our own from scratch so that we get a good feel for it.

Request Body

While many HTTP libraries will process the entire incoming request before passing control to the program, Node’s http module takes a different approach. It constructs and passes the http.IncomingMessage and http.ServerResponse objects as soon as it has received the header portion of the request. This means the body may still be being transmitted to the server as you start to process the request.

This is fine for GET requests, as they don’t have a body. But for requests that do have a body, it means you have to process the incoming data yourself.

To do so, there are three events made available in the http.IncomingMessage object: data, end, and error.

The data event occurs whenever a new chunk of the body is available, and that chunk is passed to the event listener. The end event occurs once all the chunks have been received. And the error event occurs when something goes wrong. The chunks are Buffer objects, and are always received in order.

Thus, we need to collect each chunk from each data event, and join them together into a single buffer at the end event. Thus, the basic scaffold for doing so is:

function loadBody(req, res) {
    var chunks = [];

    // listen for data events
    req.on('data', (chunk) => {
      chunks.push(chunk);
    });

    // listen for the end event 
    req.on('end', () => {
        // Combine the chunks 
        var data = Buffer.concat(chunks);

        // DO SOMETHING WITH DATA
    });

    // listen for error events 
    req.on('error', (err) => {
        // RESPOND APPROPRIATELY TO ERROR
    });
}

Note that these events occur asynchronously, so this scaffold needs to be integrated into an appropriate asynchronous processing strategy.

You might wonder why the developers of Node chose to handle request bodies in this fashion. Most likely, it was for the degree of control it offers the programmer. Consider the case where you decide your program needs to be able to stop an upload - you can do so with http.IncomingMessage.destroy(), which immediately closes the socket connecting the client to the server, effectively terminating the upload.

This can prevent the server from doing unneeded work and consuming unnecessary bandwidth. Of course, the decision to stop an upload is highly situationally dependent - thus they leave it in your hands as the programmer.

In fact, a common Denial of Service attack known as an HTTP flood often takes advantage of POST requests to submit requests with large bodies to consume server resources. By giving the programmer the ability to stop uploads this way helps counter this kind of attack.

Cookies

HTML was designed as a stateless protocol. This means that there is no expectation for the server to keep track of prior requests made to it. Each incoming HTTP request is effectively treated as if it is the first request ever made to the server.

This was an important consideration for making the web possible. Consider what would happen if our server needed to keep track of what every visitor did on the site - especially when you have thousands of unique visitors every second? You would need a lot of memory, and a mechanism to tell which user is which. That’s a lot of extra requirements to getting a web server working.

But at the same time, you probably recognize that many of the websites you use every day must be doing just that. If you use Office 365 or Google Docs, they manage to store your documents, and not serve you someone else’s when you log in. When you visit an online store and place products in your shopping basket, they remain there as you navigate around the site, loading new pages. So how are these websites providing state?

The answer that was adopted as a W3C standard - RFC 2695 is cookies. Cookies are nothing more than text that is sent along with HTTP Requests and Responses. A server can ask a client to store a cookie with a Set-Cookie header. From that point on, any request the client makes of that server will include a Cookie header with the cookie information.

It’s kind of like if your forgetful teacher gives you a note to show him the next time you come to office hours, to help remind him of what you talked about previously. This way, the server doesn’t have to track state for any visitor - it can ask the visitor’s client (their browser) to do it for them.

What is a Cookie?

Let’s dig deeper into this idea and implementation of cookies.

Much like a web request or response, a cookie is nothing more than a stream of data. It gets passed between client and server through their headers. Specifically, the Set-Cookie header sends the cookie to the client, and the client will automatically return it in subsequent requests using the Cookie header.

Say we want to establish a session id, (SID), for every unique visitor to our site. We would have our server send the SID in a Set-Cookie header:

Set-Cookie: SID=234

The browser would store this value, and on subsequent requests to the same domain, include a corresponding Cookie header:

Cookie: SID=234

Unless the Domain cookie attribute is specified (see below), a cookie is only sent to the domain that it originated from. It will not be sent to subdomains, i.e. a cookie that came from ksu.edu will not be sent to the server with hostname cs.ksu.edu.

Structure of a Cookie

As you might have guessed from the example, the string of a cookie is a set of key/value pairs using the equals sign (=) to assign the value to the key. But we can also include multiple cookie pairs using a semicolon (;) to separate the cookie pairs, i.e.:

Set-Cookie: SID=234; lang=en-US

Both cookie pairs are sent back in a Cookie header:

Cookie: SID=234; lang=en-US

We can chain multiple pairs of cookie pairs. After the last cookie pair, we need another semicolon, then any cookie attributes, also separated by semicolons. For example:

Set-Cookie: SID=234; lang=en-US; EXPIRES=Wed, 04 April 2019 011:30:00 CST

adds a cookie attribute specifying an expiration date for the cookie. The legal cookie attributes are laid out in RFC6265, but we’ll reprint them here.

Cookie Attributes

There are a limited number of defined cookie attributes, and each changes the nature of the cookie and how a browser will work with it in an important way. Let’s examine each in turn:

Expires As you might expect, this sets an expiration date for the cookie. After the date, the browser will throw away the cookie, removing it from its internal cache. It will also stop sending the cookie with requests. If you need to erase a value in a cookie you’ve already sent to a user, you can respond to their next request with a new Set-Cookie header with the Expires value set in the past. To avoid issues with time zones, the date supplied must conform to the HTTP-Date format set out in RFC1123.

Max-Age The Max-Age also sets an expiration date, but in terms of a number of seconds from the current time. A value of 0 or less (negative) will remove the cookie from the browser’s cache, and stop sending the cookie in future requests. IE 7, 8, and 9 do not support the Max-Age attribute; for all other browsers, Max-Age will take precedence over Expires.

Domain As described above, cookies are by default sent only to the domain from which they originated, and no subdomains. If the domain is explicitly set with the Domain attribute, subdomains are included.

Path The value of this attribute further limits what requests a cookie will be sent to. If the Path is specified, only requests whose url matches the value will be sent. This does include subpaths, i.e. Path=/documents will match the urls /documents/, /documents/1.doc, and /documents/recent/a.pdf.

Secure Cookies including this attribute (it does not need a value) will only be sent by the browser with HTTPS requests, not HTTP requests. Note that sending the cookie over HTTPS does not encrypt the cookie data - as a header, it is sent in the clear for both HTTP and HTTPS.

HttpOnly Cookies can normally be accessed by JavaScript on the client, from the Document.cookie property. Setting this attribute (it does not need a value) tells the browser not to make it accessible.

SameSite (in Draft) This new cookie attribute is intended to help a server assert that its cookies should not be sent with any cross-site requests originating from this page (i.e. <script> tags, <img> tags, etc. whose source is a different site, or AJAX requests against other sites). This is intended to help combat cross-site scripting attacks. As you might expect, this is only supported in the newest browser versions.

Cookies and Security

As with all web technologies, we need to consider the security implications. Cookies are received and returned by the client - and we know that anything the client is responsible for can be manipulated by an adversarial agent. Let’s think through the implications.

All cookies received by a browser and cached there. How long they are stored depends on the Max-Age and Expires attributes. Browsers are told to delete cookies older than their Max-Age, or whose Expires duration has passed. If neither of these are set, the cookie is considered a session cookie - it is deleted when the browser is closed (this is why you should always close a browser you use on a public computer).

While session cookies only exist in memory, cookies with expiration dates may be persisted as text files in the browser’s data directory. If this is the case, it is a simple matter for a knowledgeable adversary with access to the computer to retrieve them. Also remember access doesn’t necessarily mean physical access - malware running on your computer can effectively copy this data as well.

Moreover, cookies are just text - say for example you add the identity of your user with a cookie Set-Cookie: user-id=735. An adversary could sign up for a legitimate account with your application, and then manipulate their cookie to return Cookie: user-id:1. Since user ids are often incrementally generated primary keys in a database, it’s a good assumption that one or two low-numbered ones will belong to administrator accounts - and this adversary has just impersonated one!

So if we need cookies to add state to our web applications, but cookies (and therefore that state) are subject to interception and manipulation - how can we protect our users and our site?

Strategies for Making Cookies Safe(er)

Some common strategies are:

• Use HTTPS
• Store as little information in a cookie as possible
• Encrypt the cookie’s values
• Sign the cookie
• Use Short Lifespans

Use HTTPS

Using Secure HTTP helps prevent cookies from being intercepted during transit, as the request body and headers are encrypted before they move across the transport layer of the Internet. This does not protect cookies once they arrive at a user’s computer, however. Still, it is a great first step.

Storing as Little in the Cookie as Possible

A good general strategy is to store as little of importance in the cookie as possible. This also helps keep the size of a site’s cookies down, which means less traffic across the network, and less to store on a user’s computer. The RFC suggests browsers support a minimum size of 4096 bytes per cookie - but this is not a requirement.

But where do we store the session information, if not in the cookie? This is the subject of the next section, https://textbooks.cs.ksu.edu/cis526/06-dynamic-web-servers/09-sessions/. This strategy usually entails storing nothing more than a session identifier in the cookie, and storing the actual data that corresponds to that session identifier on the server. Effectively, we make our server statefull instead of stateless.

Encrypting Cookie Values

Another common strategy is encrypting the value of a cookie using a reversible encryption algorithm before sending it to the client. This kind of encryption technique is a mathematical function that can easily be reversed, provided you know the key used during the encryption process. This key should be stored in the server and never shared, only used internally to encrypt outgoing cookies and decrypt incoming ones.

This strategy only works if you don’t need to know the cookie’s value on the client (as the client will not have the key). Also, be wary of sending any details of the encryption with the encrypted data - anything that helps identify the encryption strategy or provides information like initialization vectors can help an adversary break the encryption. Also, check what algorithms are considered strong enough by current standards; this is always a moving target.

Signing Cookie Values

A third common technique is signing the cookie with a hash. A hash is the result of a one-way cryptographic function - it is easy to perform but very difficult to reverse. How it works is you hash the value of the cookie, and then send both the original data’s value and the hashed value as cookies. When you receive the cookie back from the client, you re-hash the value found in the cookie, and compare it to the hashed value in the cookie. If the two values are different, then the cookie has been modified.

Use Short Lifespans

The above strategies only keep adversaries from modifying cookies. There is still the danger of valid cookies being captured and used by an adversary - a technique known as session hijacking. Using HTTPS can prevent session hijacking from within the network, but will not protect against cookies lifted from the client’s computer.

Making sessions expire quickly can help mitigate some of the risk, especially when the adversary is using physical access to the computer to retrieve cookies - you wouldn’t expect them to do this while the user is still using the computer.

But non-physical access (such as a compromised computer) can lift a cookie while it is in use. Regenerating a session identifier (assuming you are using the first strategy) on each login can help, especially if you prompt users to log in before allowing especially risky actions. Adding some additional tracking information like the originating IP address to the session and prompting the user to validate a session by logging back in when this changes can also help.

Be aware that these strategies can also be very frustrating for users who are prompted to sign in repeatedly. As always, you need to balance security concerns with user satisfaction for the kind of activities your application supports.

Combinations of the Above

Clearly these strategies can be used in combination, and doing so will provide a better degree of safety for your clients. But the can also increase computation time, network traffic, and user frustration within your web application. Deciding just how much security your application needs is an important part of the design process.

Sessions

If HTTP is stateless, how do e-commerce websites manage to keep track of the contents of your shopping cart as you navigate from page to page? The answer is the use of a session - a technique by which state is added to a web application.

A session therefore provides a mechanism for storing information between requests, and must be unique to a specific user. All session techniques begin with a cookie, so you should be familiar with that concept.

There are three basic session techniques you will likely encounter in web development. Let’s take a look at each.

Cookie Session

In a cookie session, the session information is completely stored within the cookie. This has the benefit that no state information needs to be stored on the server, and it is easy to implement.

There are several downsides to the cookie session. The first is that it comes with a limited amount of reliable storage (the standard suggests that browsers support a minimum cookie size of 4096 bytes per cookie, but there is no hard requirement). Second, cookies are somewhat vulnerable, as they can be intercepted in-transit, and they can also be recovered from a browser’s cookie storage by a knowledgeable adversary.

In-Memory Cookies

A second approach is to store the session data on the server using some form of memory cache. A straightforward implementation for a Node server would be to use an object, i.e.:

var sessions = {}

Each time a new visitor comes to the website, a unique ID is generated, and used as the the key for their session, and sent to the client as a cookie, i.e.:

var sessionID = [some unique id];
res.setHeader("Set-Cookie", `session-id=${sessionID}; lang=en-US`);
sessions[sessionID] = {
    // session data here
}

On subsequent requests, the cookie can be retrieved from the request and used to retrieve the session data:

var cookie = req.headers["cookie"];
var sessionID = /session-id=([^;])/.match(cookie);
var session = sessions[sessionID];

Because the session data is never sent in its raw form to the client, it is more secure. However, the session can still be hijacked by a malicious user who copies the cookie or guesses a valid session id. To counter this, all requests should be updated to use secure protocol (https) and the cookie should be encrypted.

Also, in-memory sessions are lost when the server reboots (as they are held in volatile memory). They should also be cleaned out periodically, as each session consumes working memory, and if a session is more than an hour old it probably will not be used again.

Database Sessions

When a website is backed by a database (as we’ll cover soon), storing the session in that database becomes an option. Functionally, it is similar to an In-Memory session, except that the database becomes the storage mechanism. Because the database offers persistent storage, the session also could be persistent (i.e. you remain “logged on” every time you visit from the same computer). Still, database sessions typically are cleaned out periodically to keep the sessions table from growing overlarge.

Database sessions are a good choice if your website is already using a database. For a website with registered users, the session id can be the user’s id, as defined in the database schema.

Template Rendering

While skilled programmers may have chafed at the restrictions imposed by server pages, there was one aspect that came to be greatly valued - the ability to embed script directly in HTML, and have it evaluated and concatenated into the HTML text.

Template Libraries

This is where template libraries come in. A template library allows you to write your HTML content as HTML in a separate file with a special syntax to inject dynamic programming script. This is often some variation of <> or {}. This approach allows you to validate the HTML, as the script portions are ignored as unknown tags (when using <>) or as static text.

When the server is running, these templates are rendered, evaluating any code and supplying the applicable variables. This process generates the HTML snippet.

This is similar to what server pages do. However, server pages represent an integrated approach - the server page framework defined all aspects of how you could interact with the OS, the web server, and other programs. In contrast, a template library provides the option of using templates within any project - obviously for generating HTML as part of a dynamic website, but potentially for other kinds of applications as well.

Thus, a template rendering library gives us a lot of flexibility in how and when we use it. For example, in using the Embedded JavaScript template library, we could rewrite our directory listing as:

<!doctype html>
<html>
    <head>
        <title>Directory Listing</title>
    </head>
    <body>
        <h2>Directory Listing for <%= pathname %></h2>
        <div style="display: flex; flex-direction: column; padding: 2rem 0">
        <% entries.forEach((entry) =>{ %>
            <a href="<%= path.posix.join(pathname, entry) %>">
                <%= entry %>
            </a>
        <% }) %>
        </div>
        <% if (pathname !== "/") { %>
            <a href="<%= path.dirname(pathname) %>">
                Parent directory
            </a>
        <% } %>
    </body>
</html>

Notice how we can embed the JavaScript directly into the file. In a Node server, we can use this template to render html:

// Build the dynamic HTML snippets
var data = {
    pathname: pathname,
    entries: entries,
    path: path
};
ejs.renderFile("templates/directory-listing.ejs", data, function(err, html){
    // TODO: render the HTML in the string variable html
});

While this may seem like just as much work as the concatenation approach, where it really shines is the ability to combine multiple templates, separating parts of the pages out into different, reusable template files. These are typically separated into two categories based on how they are used, partials and layouts.

Partials

A partial is simply a part of a larger page. For example, the entries we are rendering in the listing could be defined in their own template file, directory-listing-entry.ejs:

<a href="<%= path %">
    <%= entry %>
</a>

And then included in the directory-listing.ejs template:

<!doctype html>
<html>
    <head>
        <title>Directory Listing</title>
    </head>
    <body>
        <h2>Directory Listing for <%= pathname %></h2>
        <div style="display: flex; flex-direction: column; padding: 2rem 0">
        <% entries.forEach((entry) =>{ %>
            <%- include('directory-listing-entry' {entry: entry, path: path.posix.join(pathname, entry) }) %>
        <% }) %>
        </div>
        <% if (pathname !== "/") { %>
            <a href="<%= path.dirname(pathname) %>">
                Parent directory
            </a>
        <% } %>
    </body>
</html>

While this may seem overkill for this simple example, it works incredibly well for complex objects. It also makes our code far more modular - we can use the same partial in many parts of our web application.

Layouts

A second common use of templates is to define a layout - the parts that every page holds in common. Consider this template file, layout.ejs:

<!doctype html>
<html>
    <head>
        <title><%= title %></title>
    </head>
    <body>
        <%- include('header') %>
        <%- content %>
        <%- include('footer') %>
    </body>
</html>

Now our directory listing can focus entirely on the directory listing:

<h2>Directory Listing for <%= pathname %></h2>
<div style="display: flex; flex-direction: column; padding: 2rem 0">
<% entries.forEach((entry) =>{ %>
    <%- include('directory-listing-entry' {entry: entry, path: path.posix.join(pathname, entry) }) %>
<% }) %>
</div>
<% if (pathname !== "/") { %>
    <a href="<%= path.dirname(pathname) %>">
        Parent directory
    </a>
<% } %>

And we can render our page using the layout:

// Build the dynamic HTML snippets
var data = {
    pathname: pathname,
    entries: entries,
    path: path
};
ejs.renderFile("templates/directory-listing.ejs", data, function(err, content){
    if(err) {
        // TODO: handle error
        return;
    }
    ejs.renderFile("templates/layout.ejs", {content: content}, function(err, html) {
        // TODO: render the HTML in the string variable html
    });
});

This layout can be re-used by all pages within our site, allowing us to write the HTML shared in common by the whole website once.

Also, while these examples show reading the template files as the response is being generated, most template engines support compiling the templates into a function that can be called at any time. This effectively caches the template, and also speeds up the rendering process dramatically.

Concise Templating Languages

Some template libraries have leveraged the compilation idea to provide a more concise syntax for generating HTML code. For example, the directory listing written using Pug templates would be:

doctype html
html(lang="en")
    head
        title= Directory Listing
    body
        h2= 'Directory Listing for ' + pathname 
        div(style="display: flex; flex-direction: column; padding: 2rem 0")
        each entry in entries
            a(href=/cis526/06-dynamic-web-servers/10-template-rendering/path.posix.join(pathname, entry)) entry 
        if pathname !== "/"
            a(href=/cis526/06-dynamic-web-servers/10-template-rendering/path.dirname(pathname))
                Parent directory

Concise templating languages can significantly reduce the amount of typing involved in creating web pages, but they come with a trade-off. As you can see from the code above, learning Pug effectively requires you to learn a new programming language, including its own iteration and conditional syntax.

Components

A newer approach, popularized by the React framework, emphasizes building components rather than pages. A component can be thought of as a single control or widget on a page. While conceptually similar to the partials described above, it organizes its structure, styling, and scripting into a cohesive and reuseable whole.

This represents a move away from the separation of concerns we described earlier. It often incorporates aspects of declarative programming rather than thinking of programming in terms of control flow. This component-based approach also lends itself to developing progressive web applications, which only download the script needed to run the portion of the app the user is interacting with at a given time.

As this approach is significantly different from the more traditional templating libraries, we’ll discuss these ideas in a later chapter.

Script Injection

Whenever we render content created by users, we open the door to script injection, a kind of attack where a malicious user adds script tags to the content they are posting. Consider this form:

<form action="post">
   <label for="comment">
      <textarea name="comment"></textarea>
    </label>
    <input type="submit"/>
</form>

The intent is to allow the user to post comments on the page. But a malicious user could enter something like:

What an awesome site <script src="http://malic.ous/dangerous-script.js"></script>

If we concatenate this string directly into the webpage, we will serve the <script> tag the user included, which means that dangerous-script.js will run on our page! That means it can potentially access our site’s cookies, and make AJAX requests from our site. This script will run on the page for all visitors!

This attack is known as script injection, and is something you MUST prevent. So how can we prevent it? There are several strategies:

1. HTML Escaping
2. Block Lists
3. Allow Lists
4. Markdown or other Markup Languages

HTML Escaping

The simplest is to “escape” html tags by replacing the < and > characters with their html code equivalents, < (less than symbol) and > (greater than symbol). By replacing the opening and closing brackets with the equivalent HTML code, the browser will render the tags as ordinary strings, instead of as HTML. Thus, the above example will render on the page as:

What an awesome site <script src=“http://malic.ous/dangerous-script.js"></script>

This can be accomplished in JavaScript with the String.prototype.replace() method:

sanitizedHtmlString = htmlString.replace(/</g, '&lt;').replace(/>/g, '&gt');

The replace() method takes either a regular expression or string to search for - and a string to replace the occurrences with. Here we use regular expressions with the global flag, so that all occurrences will be replaced. We also take advantage of the fact that it returns the modified string to do method chaining, invoking a second replace() call on the string returned from the first call.

The downside to this approach is that all html in the string is reinterpreted as plain text. This means our comments won’t support HTML codes like <b> and <i>, limiting the ability of users to customize their responses.

Allow Lists

Allow lists (also called “whitelists”) are a technique to allow only certain tags to be embedded into the page. This approach is more involved, as we must first parse the HTML string the same way a browser would to determine what tags it contains, and then remove any tags that don’t appear in our allow list.

Instead of writing your own code to do this, it is common practice to use a well-maintained and tested open-source library. For example, the NPM package sanitize-html provides this ability.

Block Lists

Block lists (also called “blacklists”) are a similar technique, but rather than specifying a list of allowed tags, you specify a list of disallowed tags. These are then stripped from html strings. This approach is slightly less robust, as new tags are added to the HTML standard will need to be evaluated and potentially added to the block list.

Markdown or other Markup Languages

A final option is to have users write their contributions using Markdown or another markup language which is transformed into JavaScript. Markdown (used by GitHub and many other websites) is probably the best-known of these, and provides equivalents to bold, italic, headings, links, and images:

# This is transformed into a h1 element 
## This is transformed into a h2 element
> This is transformed into a blockquote 
_this is italicized_ *as is this*
__this is bold__ **as is this**
* this
* is
* an 
* unordered 
* list
1. this
2. is 
3. a 
4. numbered
5. list