Legal Question in Technology Law in New York

Well, plagiarism isn't a legal term, but if someone were to use what you describe while claiming it to be one's own creation, it sounds to me like it would be deemed to be "plagiarism."

Back in the 1970's when I served in the Pennsylvania Department of Justice in Harrisburg, a lot of companies were advertising in high school and college newspapers that they could provide "term papers" that students could hand in as their own. The Attorney General decided to sue a number of them under the "consumer fraud" statute, asserting that even though the purchasers weren't defrauded when they bought the term papers, the schools were defrauded when the students turned them in as their own, and the public was defrauded when those students achieved grades higher than they would otherwise have been entitled to.

Furthermore, the program that you hypothesize about would generally use, as input, text that is copyrighted. The output would then be a "derivative work" based on the copyrighted work. The owner of the copyright has the right to prohibit others from publishing derivative works without the owner's permission.